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Executive  Summary 


The  Intelligence  Advanced  Research  Projects  Activity  (IARPA)  supports  research  programs 
to  develop  forecasting  capabilities,  including  technology  forecasting  tools,  which  aim  to  predict 
technology’s  future  characteristics  or  applications.  IARPA  requested  that  the  IDA  Science  and 
Technology  Policy  Institute  (STPI)  explore  the  current  and  potential  future  use  of  technology 
forecasts  in  science  and  technology  (S&T)  decision-making  to  help  IARPA  improve  the 
development  and  implementation  of  S&T  analytical  capabilities. 

The  project  had  two  main  goals.  The  first  was  to  understand  the  current  state  of  technology 
forecasting  in  Federal  Government  agencies  and  the  role  that  technology  forecasting  plays  in 
decision-making.  The  second  goal  was  to  understand  what  technology  forecasting  tools  are  desired 
by  Federal  agencies,  including  the  characteristics  of  technology  forecasting  tools,  approaches,  and 
information  that  personnel  would  like  to  use.  To  understand  the  current  and  potential  use  of 
technology  forecasting  tools  and  decisions  in  the  Federal  Government,  STPI  research  staff 
conducted  interviews  with  personnel  at  a  range  of  Federal  agencies  that  use  infonnation  about  new 
or  emerging  technologies  or  technology  applications. 

Use  of  Technology  Forecasting  in  the  Government 

Two  different  models  describe  when  and  why  an  agency  begins  a  technology  forecasting 
effort:  (1)  as  a  consequence  of  routine  surveillance  and  reporting  requirements  and  (2)  as  a  reaction 
to  a  specific  event  occurring  or  a  certain  need  arising.  The  specific  applications  of  technology 
forecasting  may  include: 

•  Gathering  intelligence, 

•  Identifying  threats  and  opportunities  emerging  from  potential  future  applications  of 
technologies  (e.g.,  public  health  threats  and  dual-use  technology  applications), 

•  Managing  research  portfolios, 

•  Understanding  how  future  scenarios  might  be  shaped  or  affected  by  today’s  long-tenn 
technology  investments,  and 

•  Understanding  economic  and  policy  implications  of  the  evolution  and  global 
availability  of  commercial  technologies  or  products. 

While  these  applications  of  technology  forecasting  may  differ  among  different  agencies  and 
offices,  the  type  of  decision-making  that  the  forecasting  supports  is  correlated  with  the  timescale 
for  the  forecast.  In  general,  the  agencies  and  offices  that  are  interested  in  the  state  of  technologies 
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or  technology  applications  within  5  years  are  preparing  to  respond  quickly  to  concerns  associated 
with  technologies  that  either  have  already  reached  the  later  stages  of  technology  readiness  or  have 
fast  development  timescales.  Agencies  and  offices  that  are  interested  in  the  state  of  technologies 
or  technology  applications  in  5-20  years  will  focus,  in  general,  on  making  funding  decisions.  The 
long-term  forecasters  that  STPI  researchers  interviewed  said  they  provide  infonnation  to  external 
decision-makers  to  help  develop  long-tenn  investments,  such  as  infrastructure  investments,  or 
long-term  organizational  strategies.  Based  on  the  interviews,  STPI  researchers  grouped  the  agency 
forecasting  timescales  into  three  bins:  (1)  short-tenn  forecasting,  which  looks  at  technology 
developments  expected  in  0-5  years;  (2)  mid-term  forecasting,  where  the  interest  is  in  technology 
developments  expected  in  5-20  years;  and  (3)  long-term  forecasting,  which  looks  at  technology 
developments  expected  in  20-30  years. 

Desired  Capabilities  for  Automated  Forecasting  Tools 

During  interviews,  personnel  identified  both  specific  tool  capabilities  that  could  help  with 
anticipating  technology  developments  and  related  needs  that  could  potentially  be  addressed  by 
automated  tools.  The  capabilities  that  can  be  addressed  by  automated  tools  can  be  divided  into 
three  categories:  (1)  tracking  and  summarization  capabilities,  which  give  the  user  infonnation 
about  the  cunent  state,  contextualized  by  the  past;  (2)  alert  capabilities,  which  would  flag  a  user  if 
a  pre-determined  potential  future  state  is  likely  to  occur;  and  (3)  forecasting  capabilities,  which 
provide  qualitative  information  about  a  future  state,  specifically,  within  the  category  of  forecasting 
capabilities,  users  expressed  interest  in  extrapolation  and  trend  analysis,  prediction  markets,  and 
the  ability  to  determine  the  likelihood  of  a  technology  having  dual-use  applications. 

The  different  types  of  desired  capabilities  conelate  with  the  short-,  mid-,  and  longer-tenn 
timescales  into  which  technology  forecasting  can  be  grouped.  The  tracking  and  summarization 
capabilities  give  infonnation  about  the  present  state,  prediction  markets  can  support  short-tenn 
forecasting,  extrapolation  and  trend  analysis  can  support  mid-  to  long-tenn  forecasting,  and  an 
alert  capability  could  give  infonnation  about  a  particular  future  state  on  any  of  these  timescales. 

Summary  Observations  from  the  Current  State  of  Practice  in  the  Use  of 
Technology  Forecasting  Tools  in  the  Government 

Forecasting  activities  within  the  Federal  Government  vary  in  tenns  of  the  timescales  of 
interest,  the  desired  outcome  or  infonnation  sought,  and  the  level  of  automation  currently  used  or 
being  pursued.  Potential  users  of  automated  tools  in  the  Federal  Government  point  to  the  need  for 
infonnation  within  the  context  of  their  missions  and  goals.  Given  the  variety  of  use  cases  that 
interviewed  personnel  described,  a  single  tool  or  capability  is  unlikely  to  meet  all  the  requirements. 

Despite  the  variation  in  the  types  of  potential  applications,  a  few  commonly  desired  tool 
capabilities  emerged  from  the  interviews:  (1)  that  they  present  infonnation  in  an  easy-to-visualize 
way,  (2)  that  they  are  customizable  to  a  user’s  specific  needs,  and  (3)  that  they  are  trustworthy. 
Interviewed  personnel  expressed  distrust  in  the  use  of  automated  tools,  including  technology 
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forecasting  tools.  This  distrust  was  expressed  by  respondents  both  within  and  outside  the 
Intelligence  Community  and  the  Department  of  Defense. 

Another  key  finding  from  the  interviews  was  that  capabilities  to  help  understand  new  or  state- 
of-the-art  technology  developments  in  a  field  (i.e.,  a  summarization  and  tracking)  are  seen  as  a 
greater  need  in  the  decision-making  process  than  a  forecasting  capability. 

The  focus  on  improving  awareness  of  new  technology  developments  (as  opposed  to 
infonnation  about  the  future  state  of  a  technology)  seems  to  be  a  result  of  two  factors  in 
combination:  (1)  analysts  getting  overwhelmed  by  the  volume  of  information  of  which  they  need 
to  be  aware  and  (2)  certain  technology  development  cycles  being  too  fast  for  analysts  to  keep  up 
with  the  technology.  As  a  result,  many  agencies  that  are  supposed  to  be  aware  of  emerging 
technologies  are  focusing  on  currently  available  new  technologies  rather  than  future  technologies 
or  applications. 

Tracking  and  summarization  capabilities  could  help  analysts  sort,  organize,  and  distill  large 
amounts  of  infonnation  as  well  as  keep  abreast  of  new  developments.  Alert  capabilities  could  flag 
analysts  when  an  indicator  passes  a  pre-detennined  threshold,  notifying  them  to  focus  their 
investigation  on  a  particular  issue. 

Personnel  interviewed  overwhelmingly  expressed  that  the  role  of  the  human  in  the  loop  was 
critical  to  the  use  of  forecasting  tools  in  the  decision-making  process.  An  output  from  a  tool, 
according  to  them,  would  most  likely  be  further  interpreted,  contextualized,  and  explained  by 
analysts  prior  to  being  elevated  to  the  level  of  a  decision-maker.  Therefore,  any  tool  should 
therefore  be  designed  to  aid  rather  than  supplant  the  role  of  the  analyst  and  should  not  be  operated 
in  a  stand-alone  fashion. 
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1.  Introduction 


The  Intelligence  Advanced  Research  Projects  Activity  (IARPA)  funds  research  on 
difficult  challenges  within  the  Intelligence  Community  (IC).  IARPA  has  undertaken  two 
science  and  technology  (S&T)  forecasting  programs:  (1)  the  Foresight  and  Understanding 
from  Scientific  Exposition  (FUSE)  program,  which  “seeks  to  develop  automated  methods 
that  aid  in  the  systematic,  continuous,  and  comprehensive  assessment  of  technical 
emergence  using  publicly  available  infonnation  found  in  published  scientific,  technical  and 
patent  literature”1  and  (2)  the  Forecasting  Science  and  Technology  (ForeST)  program, 
which  seeks  to  “develop  and  test  methods  for  generating  accurate  forecasts  for  significant 
[S&T]  milestones,  by  combining  the  judgements  of  many  experts.”2  To  help  IARPA 
improve  the  development  and  implementation  of  its  S&T  analytical  capabilities,  IARPA 
requested  that  the  IDA  Science  and  Technology  Policy  Institute  (STPI)  explore  the  current 
and  potential  future  use  of  technology  forecasts  in  S&T  decision-making. 

A.  Goals 

The  project  had  two  main  goals.  The  first  goal  was  to  understand  the  current  state  of 
technology  forecasting  in  Federal  agencies  in  terms  of  the  tools  used,  the  needs  supported 
by  those  tools,  and  the  role  technology  forecasting  plays  in  decision-making.  The  second 
goal  was  to  understand  what  technology  forecasting  capabilities  are  desired  by  personnel 
in  Federal  agencies  in  terms  of  the  characteristics  of,  approaches,  and  infonnation  that 
agencies  would  like  to  use,  and  the  level  of  tolerance  for  the  uncertainty  involved  in 
technology  forecasts. 

B.  Definitions 

To  scope  the  findings  of  the  report,  it  is  necessary  to  define  technology  and  technology 
forecasting.  Our  definition  of  technology  is  derived  from  the  Department  of  Defense 
(DOD)  Technology  Readiness  Assessment  and  the  National  Aeronautics  and  Space 
Administration  (NASA)  Technology  Readiness  Levels  (TRLs),  which  assume  a  nine-step 
process  of  development  beginning  with  basic  research  and  culminating  in  an  operational 
system.  The  first  step  is  observation  and  reporting  of  basic  principles.  In  the  second  step, 


1  IARPA  website,  “Foresight  and  Understanding  from  Scientific  Exposition  (FUSE),” 
http://www.iarpa.gov/index.php/research-programs/fuse. 

2 

'  IARPA  website,  “Forecasting  Science  &  Technology  (ForeST),” 
http://www.iarpa.gov/index.php/research-programs/forest. 
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technology  concepts  or  applications  are  formulated.  The  third  through  the  sixth  steps 
include  experimentation  and  validation  of  a  technology.  The  seventh  and  eighth  steps  are 
technology  demonstration  steps,  and  the  final  step  is  full  proof  that  the  technology  is  able 
to  achieve  its  mission.  The  concept  of  technology  forecasting  is  related  to  this  assumed 
development  pathway,  because  the  idea  behind  technology  forecasting  is  that  advanced 
technologies  have  precursors  that  could  be  detected  and  evaluated. 

Our  definition  of  technology  is  thus  the  result  of  activities  ranging  from  the 
speculative  application  of  basic  principles  observed  through  scientific  research  to  the 
development  of  proven  systems  (Assistant  Secretary  of  Defense  for  Research  and 
Engineering  2011;  Mankins  1995). 

We  found  no  single,  widely  accepted  definition  for  technology  forecasting.  A  report 
on  technology  forecasting  by  the  National  Research  Council  (NRC)  defines  it  as  the 
“prediction  of  the  invention,  timing,  characteristics,  dimension,  performance,  or  rate  of 
diffusion  of  a  machine,  material,  technique,  or  process  serving  some  useful  purpose”  (NRC 
2010).  In  a  report  by  The  Tauri  Group,  technology  development  forecasts  are  described  as 
including  “forecasts  for  the  emergence  of  new  technologies,  the  evolution  of  existing 
technology,  or  the  migration  of  technologies  to  new  application  areas...”  (Mullins  2012). 
In  an  early  paper  on  technology  forecasting,  J.  G.  Wissema  states  that  “technology 
forecasting  studies  are  systematic  investigations  into  the  future  development  and 
application  of  technologies.  It  is  not  so  much  that  one  wants  to  predict  the  future  but  rather 
one  wants  to  see:  which  interactions  exist  with  other  developments;  which  actions  are 
possible  and  what  effects  they  will  have”  (Wissema  1982).  For  this  report,  technology 
forecasting  is  defined  as  the  prediction  of  a  technology’s  future  characteristics  or 
applications,  most  often  bounded  within  a  particular  window  of  time. 

C.  Approach 

To  understand  the  use  of  technology  forecasting  tools  and  decisions  in  the  Federal 
Government,  STPI  research  staff  conducted  interviews  with  personnel  at  a  range  of  Federal 
agencies  involved  in  using  infonnation  about  new  or  emerging  areas  of  research  and 
technology  or  new  or  emerging  applications  for  research  and  technology.3  This  report  does 
not  include  interviews  with  individuals  at  private  sector  technology  forecasting  companies 
because  IARPA’s  goal  is  to  develop  technology  forecasting  tools  for  the  Federal 
Government,  and  therefore  is  most  interested  in  the  views  of  their  potential  customers. 
Interviews  were  held  with  personnel  within  DOD  and  the  IC,  as  well  as  within  offices 
involved  in  other  areas  of  science  and  technology,  such  as  healthcare,  biomedical  research, 


3 

Although  some  of  the  affiliations  of  the  personnel  interviewed  are  departments,  offices,  or  some  other 
entity  within  the  Federal  Government,  we  refer  to  them  all  as  “Federal  agencies”  or  “agencies” 
throughout  this  report. 
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energy  research,  and  export  control.  A  diverse  set  of  agencies  was  chosen  to  explore  a 
range  of  technology  forecasting  needs  and  applications.  They  are  as  follows: 

•  Central  Intelligence  Agency,  Directorate  of  Science  and  Technology 

•  Defense  Threat  Reduction  Agency,  Research  and  Development  Directorate 

•  Department  of  the  Air  Force,  Air  Force  Office  of  Scientific  Research 

•  Department  of  Commerce,  Bureau  of  Industry  and  Security,  Emerging 
Technology  and  Research  Advisory  Committee  and  Office  of  Technology 
Evaluation 

•  Department  of  Defense,  Assistant  Secretary  of  Defense  for  Research  and 
Engineering,  Office  of  Technical  Intelligence 

•  Department  of  Health  and  Human  Services,  Agency  for  Healthcare  Research 
and  Quality 

•  Department  of  the  Navy,  Chief  of  Naval  Operations  Strategic  Studies  Group 

•  Energy  Infonnation  Administration 

•  Government  Accountability  Office 

•  National  Institutes  of  Health,  National  Institute  of  Biomedical  Imaging  and 
Bioengineering 

•  Office  of  the  Director  of  National  Intelligence,  Acquisition,  Technology,  and 
Facilities 

Appendix  A  provides  the  interview  method  and  guide.  The  infonnation  from  these 
interviews  (summarized  in  Appendix  B)  was  supplemented  with  additional  research  and 
case  studies. 

D.  Report  Overview 

The  structure  of  this  report  is  as  follows: 

•  In  Chapter  2,  we  discuss  the  current  use  of  technology  forecasting  in  the  Federal 
Government,  including  triggers  that  initiate  technology  forecasts,  forecasting 
timescales  that  different  agencies  examine,  current  and  potential  use  of 
technology  forecasts  in  decision-making,  and  current  levels  of  automation  in 
technology  forecasting.  We  also  discuss  the  issue  of  trust  in  forecasting  tools. 

•  In  Chapter  3,  we  discuss  the  capabilities  that  the  personnel  we  interviewed 
currently  do  not  have  but  would  like  to  have. 

•  In  Chapter  4,  we  discuss  the  implications  of  our  findings  for  IARPA’s 
technology  forecasting  efforts  and  provide  recommendations. 
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•  Appendix  A  contains  the  interview  methodology  and  guide;  Appendix  B 

contains  summaries  of  the  interviews;  Appendix  C  contains  descriptions  of  some 
of  the  analytical  and  forecasting  tools,  techniques,  and  other  aids  discussed  in 
the  report;  and  Appendix  D  provides  case  studies  of  agencies  that  could 
potentially  use  technology  forecasting  tools  to  support  their  different  missions. 
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2.  Current  State  of  Technology  Forecasting  in  the 

Federal  Government 


During  interviews,  respondents  were  not  presented  with  a  definition  of  technology 
forecasting  because  part  of  the  goal  of  the  interviews  was  to  understand  the  respondents’ 
own  perceptions  and  understanding  of  the  tenn.  Respondents  were  informed  that  the  goal 
of  the  project  was  to  understand  current  and  potential  roles  of  technology  forecasting  and 
technology  forecasting  tools  in  decision-making  processes  in  the  Federal  Government,  and 
then  they  were  asked  whether  they  currently  use  technology  forecasting  or  projections. 

Respondents  seemed  to  be  comfortable  speaking  about  technology  forecasting  despite 
not  being  given  a  definition.  The  majority  of  respondents  viewed  awareness  about  potential 
future  directions  of  research  and  technology  to  be  a  part  of  their  jobs.  In  addition  to 
speaking  about  technology  forecasting,  they  often  spoke  about  tracking  people  or  events. 
Respondents  frequently  spoke  about  science  and  technology  analytics  that  are  distinct  from 
forecasting.  The  interviews  revealed  that  technology  forecasting  is  relevant  for  a  range  of 
mission-specific  goals,  including: 

•  Gathering  intelligence; 

•  Identifying  threats  and  opportunities  emerging  from  potential  future  applications 
of  technologies  (e.g.,  public  health  threats  and  dual-use  technology 
applications); 

•  Managing  research  portfolios, 

•  Understanding  how  future  scenarios  might  be  shaped  or  affected  by  today’s 
long-tenn  technology  investments,  and 

•  Understanding  economic  and  policy  implications  of  the  evolution  and  global 
availability  of  commercial  technologies  or  products. 

Table  2  summarizes  the  relationships  that  the  interviewed  personnel  have  to 
technology  forecasting  at  their  agencies. 


5 


Table  2.  Interview  Respondents’  Agency  Affiliation,  Goal,  and 
Relationship  to  Technology  Forecasting 


Respondents’  Agency  Affiliation 

Central  Intelligence  Agency  (CIA),  Directorate  of 
Science  and  Technology  (DS&T) 

Department  of  the  Air  Force,  Air  Force  Office  of 
Scientific  Research  (AFOSR) 

Department  of  Commerce  (DOC),  Bureau  of  Industry 
and  Security  (BIS)  Emerging  Technology  and 
Research  Advisory  Committee  (ETRAC)  and  Office  of 
Technology  Evaluation  (OTE) 

Department  of  Defense  (DOD),  Assistant  Secretary  of 
Defense  for  Research  and  Engineering  (ASD[R&E]), 
Office  of  Technical  Intelligence  (OTI) 

Department  of  Health  and  Human  Services  (HHS), 
Agency  for  Healthcare  Research  and  Quality  (AHRQ) 

Department  of  the  Navy,  Chief  of  Naval  Operations 
(CNO),  Strategic  Studies  Group  (SSG) 

Defense  Threat  Reduction  Agency  (DTRA),  Research 
and  Development  Directorate  (J9) 

Energy  Information  Administration  (EIA) 


Agency  Goal 

DS&T  researches,  develops,  and  applies  advanced 
technologies  that  provide  the  national  with  a 
significant  intelligence  advantage3 

AFOSR  funds  research  that  supports  the  Air  Force 
goals  of  control  and  maximum  utilization  of  air,  space, 
and  cyberspaceb 

BIS  determines  export  controls  needed  for  “dual-use” 
technologies,  or  technologies  that  have  both  civil 
application  and  applications  related  to  the  military, 
terrorism,  or  weapons  of  mass  destruction0 

OTI  analyzes  global  science  and  technology  (S&T) 
activities  to  inform  research  investments'11 

AHRQ  informs  patient-centered  outcomes  research 
investments® 

CNO  SSG  generates  revolutionary  naval  warfare 
concepts f 

DTRA  J9  funds  and  conducts  research  and 
development  for  threat  reduction9 

EIA  produces  short-term  forecasts  and  long-term 
projections  of  energy  sources,  end  uses,  and  energy 
flows  h 


Agency  Relationship  to  Technology  Forecasting 

DS&T  needs  to  be  aware  of  new  technologies  that  will 
help  to  maintain  a  significant  intelligence  advantage 

AFOSR  needs  to  be  aware  of  emerging  areas  of 
research 

BIS  identifies  emerging  dual-use  technologies 


OTI  conducts  technology  watch  and  horizon  scanning 
activities.  They  are  also  pursuing  automated 
approaches  for  these  activities01 

AHRQ  has  developed  the  Healthcare  Horizon 
Scanning  System,  which  scans  emerging  healthcare 
technologies 

CNO  SSG  needs  to  be  aware  of  future  technologies 
and  their  implications  for  the  Navy 

DTRA  J9  needs  to  be  aware  of  new  technologies  to 
tackle  existing  threats  and  needs  to  be  prepared  for 
emerging  and  future  threats 

EIA  models  energy  and  economic  trends 


Respondents’  Agency  Affiliation  Agency  Goal  Agency  Relationship  to  Technology  Forecasting 


Government  Accountability  Office  (GAO) 


National  Institutes  of  Health  (NIH),  National  Institute 
of  Biomedical  Imaging  and  Bioengineering  (NIBIB) 


Office  of  the  Director  of  National  Intelligence  (ODNI), 
Acquisition,  Technology,  and  Facilities  (ATF) 


GAO  prepares  technology  assessments  of  current 
and  emerging  technologies  to  understand 
implications,  challenges  and  opportunities  for  the 
Federal  Government  and  their  potential  societal 
impacts.1 

NIBIB  funds  the  research  and  development  of  new 
biomedical  imaging  and  bioengineering  techniques 
and  devices  to  improve  disease  detection,  prevention 
and  treatment.-1 

ATF  informs  research  and  development  investments 

1/ 

to  solve  current  and  future  intelligence  challenges. 


GAO  needs  to  be  aware  of  the  societal  implications  of 
current  and  emerging  technologies 


NIBIB  needs  to  be  aware  of  new  and  emerging  areas 
of  research  and  technology  in  this  field 


ATF  needs  be  aware  of  emerging  science  and 
technology  and  the  potential  future  capabilities  that 
they  could  offer  for  the  intelligence  community. 


a  CIA  Science  and  Technology  Office  Website,  ‘Who  We  Are,”  https://www.cia.gov/offices-of-cia/science-technology/who-are-we.html. 
b  AFOSR  website,  “About — Mission,”  http://www.wpafb.af.mil/library/factsheets/factsheet.asp?id=9492. 

c  Emerging  Technology  and  Research  Technical  Advisory  Committee  website,  “U.S.  Department  of  Commerce  Charter  of  the  Emerging  Technology  and  Research 
Advisory  Committee,”  http://www.bis.doc.gov/index.php/licensing/28-technology-evaluation/147-emerging-technology-and-research-technical-advisory-committee. 
d  Department  of  Defense  Research  &  Engineering  Enterprise  website,  “Science  &  Technology  Corner,  Technology  Watch  and  Horizon  Scanning  for  the  Department  of 
Defense,”  http://www.acq.osd.mil/chieftechnologist/cto/cto_TWHS.html. 

e  AHRQ,  Effective  Health  Care  Program  website,  “AHRQ  Healthcare  Horizon  Scanning  System,”  http://effectivehealthcare.ahrq.gov/index.cfm/who-is-involved-in-the- 
effective-health-care-programl/ahrq-horizon-scanning-system/. 
f  CNO  SSG  website,  “Overview — About,”  https://usnwc.edu/About/Chief-Naval-Operations-Strategic-Studies-Group.aspx. 

9  DTRA  website,  “Research  &  Development,”  http://www.dtra.mil/Research.aspx. 
h  EIA  website,  “About  EIA — Mission  and  Overview,”  http://www.eia.gov/about/mission_overview.cfm. 

1  “Technology  Assessment”  URL:  http://www.gao.gov/technology_assessment/key_reports. 

J  “National  Institute  of  Biomedical  Imaging  and  Bioengineering  (NIBIB) — Mission,”  http://www.nih.gov/about-nih/what-we-do/nih-almanac/national-institute-biomedical- 
imaging-bioengineering-nibib. 
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ODNI  website,  “Acquisition,  Technology,  &  Facilities — What  We  Do,”  http://www.odni.gov/index.php/about/organization/acquisition-technology-and-facilities-what-we-do. 


To  fully  explore  this  chapter’s  topic  of  the  current  state  of  technology  forecasting,  we 
discuss  triggers  that  initiate  or  prompt  technology  forecasting  activities;  different 
applications  of  technology  forecasting,  which  can  be  divided  into  the  timescales  for  which 
the  forecasts  are  desired;  the  decision-making  structures  in  which  forecasts  are  used;  the 
variation  in  the  amount  and  type  of  automation  in  forecasting;  and  issues  of  trust 
surrounding  the  use  of  automated  forecasts. 

A.  Triggers  that  Initiate  Technology  Forecasting 

Two  general  models  describe  when  an  investigation  of  a  new  or  emerging  technology 
or  application  is  initiated.  The  first  model  is  when  the  need  for  a  technology  or  application 
is  initiated  by  routine  surveillance  and  reporting  requirements.  Some  technology  forecasts 
are  made  periodically  because  of  an  agency’s  mission.  Examples  include  the  Annual 
Energy  Outlook  produced  by  the  Energy  Information  Administration  (EIA),  and  the 
Potential  High  Impact  Reports  produced  by  the  Agency  for  Healthcare  Research  and 
Quality  (AHRQ).  These  reports  are  updated  frequently  (yearly  for  the  Annual  Energy 
Outlook  and  every  6  months  for  the  Potential  High  Impact  Reports),  and  the  updates  are 
independent  of  any  particular  event  occurring. 

The  second  model  is  a  reactive  model  in  that  a  technology  or  application  is  examined 
to  understand  the  potential  impact  of  a  specific  event  occurring  or  need  arising.  A  basic 
research  development,  a  new  commercial  product,  or  a  dramatic  decrease  in  cost  for  a 
technology  could  trigger  an  investigation  into  the  current  and  anticipated  state  of  a 
technology  or  a  technology-influenced  event  to  detennine  the  various  impacts  of  these 
developments.  Another  trigger  could  be  the  acquisition  of  companies  by  other  companies, 
whether  it  is  a  single  company  buying  many  smaller  companies  or  companies  in  one 
country  purchasing  many  companies  of  a  particular  industry  in  another  country.  These  two 
models  (routine  surveillance  versus  a  reaction)  are  similar  to  what  is  referred  to  as  a  police- 
patrol  model  versus  a  fire-alarm  model  in  the  context  of  congressional  oversight 
(McCubbins  and  Schwartz  1984). 

B.  Technology  Forecasting  Applied  to  Decision-Making  at 
Different  Timescales 

Based  on  their  function  and  goals,  agencies  of  the  Federal  Government  may  be 
interested  in  understanding  developments  at  different  points  in  a  technology’s  evolution. 
For  example,  defense  acquisition  programs  makes  technology  investments  in  the  20-30- 
years  before  the  expected  delivery  of  a  platform.  These  agencies  are  interested  in 
understanding  basic  research  areas  or  very  early  stage  technologies  whose  desired 
applications  are  anticipated  to  be  developed  for  military  use  in  the  20-30-year  time  frame. 

On  the  other  hand,  agencies  that  align  more  closely  with  a  reactive  agenda,  such  as 
the  Department  of  Commerce  (DOC)  Bureau  of  Industry  and  Security  (BIS),  which  makes 
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decisions  on  export  controls,  and  the  Defense  Threat  Reduction  Agency  (DTRA)  Research 
and  Development  Directorate  (J9),  whose  goal  is  to  identify  threats  associated  with  a  new 
or  unanticipated  use  of  a  technology,  are  more  interested  in  understanding  technologies 
that  have  reached  (or  are  close  to)  the  application  stage,  and  may  be  used  in  ways  that  have 
an  impact  on  national  security.  These  agencies  are  interested  in  understanding  research 
developments  or  commercially  available  technologies  that  may  arise  in  the  0-5-year  time 
frame. 

The  more  long-term  the  investment,  the  greater  the  need  to  look  upstream  of  the 
technology  development  process  and  track  any  changes  therein;  agencies  that  are  interested 
in  technologies  when  they  are  much  closer  to  the  point  of  application  could  incorporate 
data  such  as  market  reports,  healthcare  trends,  or  product  specifications,  as  input  into  their 
technology  forecasting  process.  Accordingly,  the  agencies  with  which  the  personnel 
interviewed  were  affiliated  are  grouped  by  their  forecasting  timescale,  and  divided  into 
three  bins:  short-term  (0-5  years  in  the  future),  mid-term  (5-20  years  in  the  future),  and 
long-tenn  (20-30  years  in  the  future).  The  timescale  bins  do  not  specify  whether  a 
technology  is  in  research  phase,  proof-of-concept  phase,  emergent,  or  widely  available. 
This  is  because  the  timescale  bins  reflect  the  time  horizon  over  which  an  agency  is 
concerned,  not  the  amount  of  time  with  respect  to  a  technology  reaching  a  certain  point  in 
its  development.  Depending  on  its  mission,  an  agency  may  be  concerned  with  any  state  of 
technology  development.  The  forecasting  efforts  within  each  timescale  bin  could  be  further 
categorized  as  to  whether  or  not  the  agency  is  using  or  pursuing  any  automated  methods 
for  their  forecasts.  The  different  levels  of  automation  are  discussed  in  section  C  of  this 
chapter. 
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1.  Short-Term  Forecasting  (0-5  years) 

In  general,  the  agencies  that  are  interested  in 
the  state  of  technologies  or  technology  applications 
that  are  within  0-5  years  of  being  realized  are 
preparing  to  respond  quickly  to  technologies  that 
have  already  reached  the  later  stages  of  technology 
readiness.  For  example,  the  Bureau  of  Industry  and 
Security  (BIS)  examines  technologies  that  are  ready 
for  commercialization  in  order  to  determine  whether 
their  export  should  be  controlled.  The  Government 
Accountability  Office  (GAO)  examines  and  briefs 
Congress  on  emerging  technologies  and  their 
potential  implications.  AHRQ  examines  procedures, 
medications,  and  other  factors  that  are  1  to  2  years 
away  from  Food  and  Drug  Administration  (FDA) 
approval  in  order  to  provide  infonnation  about 
healthcare  coverage.  DTRA  J9  initiates  research 
efforts  in  response  to  emerging  threats.  The  Central 
Intelligence  Agency  (CIA)  Directorate  of  Science 
and  Technology  examines  technologies  for  science 
and  technology  intelligence  purposes.  The 
directorate’s  forecasting  efforts  look  at  technologies 
that  can  be  acquired  within  3  years,  which 
corresponds  to  their  acquisition  cycle. 

Because  0-5-year  technology  forecasts  appear  to  support  fields  that  are  rapidly 
changing  and  fields  that  need  to  be  aware  of  later-stage  technology  developments,  it  is 
possible  to  use  these  criteria  to  identify  additional  agencies  that  could  use  short-term 
forecasts.4  For  those  agencies  that  are  interested  in  late-stage  technology  development, 
information  summarization  and  horizon  scanning  tools  could  help  since  strong  indicators 
of  the  technology  should  already  be  present.  For  the  agencies  that  are  interested  in  rapidly 
changing  fields,  any  tool  would  have  to  be  able  to  take  in  new  information  frequently 
enough  to  provide  useful  insights  on  likely  future  developments. 


Is  It  Short-Term  Forecasting  or 
Is  It  “Nowcasting?” 

In  many  interviews,  respondents 
were  concerned  about  being  able 
to  “keep  up”  with  new  technologies 
or  applications  in  their  fields. 
Interview  respondents  were  more 
interested  in  capabilities  to  help 
them  understand  new  or  state-of- 
the-art  technology  developments 
in  their  field  (i.e.,  a  summarization 
capability  that  improves  their 
situational  awareness)  than 
provide  information  about  a  future 
state  of  a  technology  or  research 
area  (i.e.,  a  forecasting  capability). 
For  the  purposes  of  this  report,  we 
consider  these  respondents’ 
agencies  and  offices  to  be 
interested  in  short-term  forecasting 
because  the  decisions  they  are 
trying  to  make  focus  on 
preparedness  for  a  future  state. 
Such  a  summarization  capability  is 
seen  as  a  greater  need  than  a 
forecasting  capability  in  the 
decision-making  process. 


4  Because  the  timescale  of  cyberthreat  and  software  development  is  frequently  less  than  5  years,  cyber- 
related  decision-makers  would  likely  also  be  concerned  with  forecasting  within  this  short  timescale. 
Similarly,  the  FDA  regulates  and  supervises  many  factors  related  to  public  health  and  would  likely  also 
want  to  be  aware  of  new  drugs,  technologies,  and  treatments  that  are  nearing  the  stage  where  testing  and 
regulation  will  be  necessary. 
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2.  Mid-Term  Forecasting  (5-20  years) 

Among  the  set  of  agencies  interviewed,  the  agencies  that  are  interested  in  technology 
developments  or  applications  that  are  expected  to  come  to  fruition  in  5-20  years  are  making 
funding  decisions.  Each  of  the  scientific  funding  agencies  in  which  interviews  were 
conducted — the  Air  Force  Office  of  Scientific  Research  (AFOSR),  the  Office  of  Technical 
Intelligence  (OTI)  in  the  Office  of  ASD(R&E),  DTRA  J9,  and  the  National  Institute  of 
Biomedical  Imaging  and  Bioengineering  (NIB IB) — make  decisions  about  what  research  to 
fund  based  on  where  they  envision  a  field  of  research  being  in  5-20  years. 

Funding  decisions  take  different  forms  and  can 
occur  at  different  levels  in  the  Federal  Government. 
Each  of  the  program  managers  interviewed  make 
choices  about  which  principal  investigators  and 
which  projects  to  fund  within  a  research  area;  higher 
level  managers  can  make  decisions  about  which 
research  directions  should  be  pursued;  and  both 
agency  leadership  and  Congress  make  large-scale 
decisions  about  new  areas  of  research  to  fund  and 
areas  of  research  funding  that  should  be  redirected 
or  discontinued. 

Funding  decisions  are  informed  by  the 
decision-maker’s  knowledge  of  the  field  of  research, 
the  mission  and  needs  of  the  relevant  agency,  and  the 
decision-maker’s  knowledge  of  the  agency’s  needs.  Technology  forecasts  could  therefore 
be  used  to  ensure  that  decisions  are  made  based  on  (1)  the  best  available  infonnation  and 
(2)  the  agency’s  mission  and  needs.  Both  of  these  needs  are  discussed  in  more  detail  below. 

Program  managers  may  not  be  aware  of  all  the  relevant  infonnation  necessary  to  make 
the  best  funding  decision.  The  large  volume  of  relevant  literature,  product  releases,  patents, 
and  other  written  material,  coupled  with  the  prohibitive  costs  of  attending  conferences 
means  that  program  managers  are  probably  going  to  see  only  a  subset  of  literature  and 
attend  only  a  subset  of  conferences.  Consequently,  a  program  manager  may  be 
inadvertently  biased  by  the  limited  information  to  which  he  or  she  is  exposed.  A  tool  that 
ingests  a  varied  and  updated  set  of  scientific  publications  and  conference  abstracts; 
identifies  trends;  and  predicts  which  areas  of  research  may  soon  grow  in  popularity,  change 
directions,  or  intersect  with  another  area  of  research  could  address  this  problem. 


Tracking  Researchers 

A  capability  to  identify  researcher 
networks  and  track  young  scientist 
collaborations  could  be  used  to 
identify  potential  areas  of  research 
and  researchers  to  fund.  One 
respondent  pointed  out  that 
traditional  metrics,  such  as  number 
of  publications  or  H-index,  which 
are  used  to  evaluate  principal 
investigators  do  not  necessarily 
apply  to  young  investigators.  A  tool 
that  is  able  to  predict  which  young 
investigators  are  likely  to  advance 
the  field  could  help  program 
managers  decide  which  young 
investigators  to  fund. 


11 


As  mentioned  previously,  technology 
requirements — such  as  DOD’s  acquisition 
requirements — are  sometimes  written  for 
programs  that  span  a  time  frame  of  10-20  years. 

The  platfonn  in  question  may  require  capabilities 
that  are  still  in  development  at  the  time  that  the 
requirements  are  being  written.  A  useful  tool 
from  the  perspective  of  acquisition  officials 
would  be  one  that  first  uses  subject  matter 
experts  to  break  down  a  desired  future 
technology  to  its  enabling  capabilities,  and  then 
uses  an  automated  approach  to  find  publications 
that  either  mention  these  capabilities  or  connect 
in  some  way  with  research  towards  these 
capabilities.  Such  a  tool  could  help  a  program 
manager  identify  where  investments  should  be 
made  in  areas  of  research  or  researchers. 

Based  on  the  findings  about  the  decision¬ 
making  structures  (discussed  in  detail  in  Chapter 
3),  automated  capabilities  of  the  types  mentioned 
above  would  likely  be  used  by  program  managers  to  identify  and  propose  new  research 
directions,  which  could  then  be  communicated  to  agency  leadership  or  Congress.  There 
may  be  other  customers  of  mid-term  technology  forecasts,  though  with  the  exception  of 
EIA,  all  of  the  respondents  concerned  with  mid-term  forecasts  worked  at  funding  agencies. 


Forecasting  versus  Prediction 
versus  Projection 

In  some  communities,  such  as  the 
climate  research  community,  the  term 
forecast  (which  is  used 
interchangeably  with  the  term 
prediction),  is  distinguished  from  the 
term  projection.  A  forecast  or 
prediction  depends  on  the  initial 
conditions,  whereas  a  projection 
depends  on  initial  conditions  and 
additional  assumptions  that  influence 
the  future  state  (Intergovernmental 
Panel  on  Climate  Change,  “Glossary 
of  Terms  used  in  the  IPCC  Fourth 
Assessment  Report,”  Working  Group 
I).  Because  projections  tend  to  be  for 
a  longer  time  scale  than  forecasts  or 
predictions,  additional  assumptions 
need  to  be  made,  so  the  projections 
tend  to  have  higher  levels  of 
uncertainty.  However,  in  this  report, 
the  terms  long-term  forecasts  and 
projections  are  used  interchangeably. 


3.  Long-Term  Forecasting  (20-30  years) 

The  EIA  makes  forecasts  for  20-30-year  timescales.  While  information  about  an 
energy  technology’s  development  may  be  used  to  develop  its  model,  EIA’s  goal  is  not  to 
forecast  technology,  but  to  forecast  energy  sources,  end-uses,  and  energy  flows.  EIA 
provides  infonnation  to  external  decision-makers  to  help  develop  long-term  investment  or 
organizational  strategies.  Because  these  forecasts  are  helpful  for  making  long-term 
infrastructure  investments,  additional  agencies  that  make  similar  large-scale,  long-term 
investments,  such  as  the  Department  of  Defense,  Department  of  Transportation,  the 
Department  of  Energy,  and  NASA,  may  also  be  interested  in  long-tenn  forecasting. 

Because  of  their  long  timescales,  these  forecasts  have  significant  levels  of  associated 
uncertainty.  It  could  be  useful  for  decision-makers  to  know  what  technology  outcomes  may 
come  to  fruition  at  a  certain  point  in  the  future,  given  a  certain  set  of  assumptions.  Trend 
analysis  tools  could  help  long-term  forecasters  by  allowing  them  to  select  different 
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parameters,  such  as  speed  of  development  or  different  manufacturing  cost  curves,  and 
examine  how  those  different  parameters  impact  a  technology’s  diffusion  or  cost. 

Another  factor  associated  with  long  timescales  of  technology  development  is  that  the 
tenninology  associated  with  a  research  area  or  technology  may  change  over  its  life  cycle. 
It  is  even  possible  for  the  name  of  a  technology  or  terms  associated  with  it  to  disappear 
from  literature  as  it  moves  through  different  stages  of  development.  Approaches  and  tools 
for  tracking  changes  in  terminology  would  be  helpful  to  users  who  track  technology  over 
a  long  period  of  time. 

C.  Decision-Making  Structures 

STPI  researchers  were  asked  to  examine  how  technology  forecasting  tools  are 
currently  used,  or  could  be  used,  in  the  decision-making  process.  In  particular,  IARPA 
expressed  interest  in  knowing  whether  information  from  technology  forecasts  directly 
detennine  a  decision,  or  whether  forecasts  are  used  as  a  source  of  background  information. 

In  discussing  how  technology  forecasts  are  used  or  could  be  used  in  the  Federal 
Government,  the  interviews  revealed  a  small  set  of  decision-making  structures  and  roles. 
STPI  research  staff  did  not  encounter  any  cases  in  which  the  decision-maker  is  the  same 
person  who  interacts  closely  with  a  technology  forecasting  tool.  The  simplest  structure 
included  two  roles:  (1)  someone  who  collects  data,  creates  or  uses  a  forecast,  and  analyzes 
it  and  (2)  the  decision-maker.  In  more  complicated  structures,  the  role  of  the  analyst  is 
separated  from  that  of  the  data  collector  or  forecast  user,  creating  three  roles:  (1)  someone 
who  collects  data  and  creates  or  uses  a  forecast,  (2)  an  analyst,  and  (3)  a  decision-maker. 
In  more  elaborate  structures,  there  are  multiple  levels  of  decision-makers.  As  an  example, 
a  team  could  include  (1)  someone  who  collects  data  and  creates  or  uses  a  forecast,  (2)  an 
analyst,  (3)  a  lower  level  decision-maker,  and  (4)  a  higher  level  decision-maker.  In  each  of 
the  steps  following  the  technology  forecast  step,  additional  infonnation  may  be  brought  in 
to  be  used  in  combination  with  the  technology  forecast  to  enable  a  decision.  These 
decision-making  structures  are  shown  in  Figure  1. 
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Two-level  structure  Three-level  structure 


Four-level  structure 


Figure  1.  Decision-Making  Structures  for  Technology  Forecasting 

The  data  collected  may  come  in  the  form  of  a  tool  output,  a  curated  data  set,  or 
information  from  subject  matter  experts  (SMEs).  SMEs  were  encountered  in  many  of  the 
decision-making  structures  and  in  all  of  the  different  forecasting  timescales.  SMEs  could 
act  either  as  primary  data  sources,  or  as  support  to  analysts  by  providing  additional 
expertise  or  context.  The  role  of  SMEs  ranged  from  providing  occasional  advice  to  being 
a  part  of  a  structured  process,  which  was  the  case  at  AHRQ  (discussed  further  in  Appendix 
D). 

The  decision-makers  that  were  discussed  in  interviews  included  program  managers, 
mid-level  managers,  and  agency  leadership.  In  many  cases,  the  decision-makers  were 
external  to  the  agency.  External  decision-makers  may  include  members  of  Congress,  the 
general  public,  and  the  private  sector.  For  example,  the  EIA  produces  the  Annual  Energy 
Outlook,  which  examines  how  various  scenarios  might  impact  the  energy  market  in  the 
coming  decades.  ( Annual  Energy  Outlook  2015  projects  out  until  2040.)  EIA  makes 
decisions  regarding  how  to  build  the  models  and  scenarios.  Based  on  the  results  of  the 
projections,  investors  may  decide  to  invest  in  different  energy  sources,  members  of 
Congress  may  use  the  projections  to  inform  policies,  and  energy  suppliers  may  make  plans 
for  infrastructure  changes.  Even  when  the  decision-makers  are  in  the  same  agency  where 
the  forecast  is  produced,  the  interview  respondents  indicated  that  technology  forecasting  is 
viewed  as  a  way  to  maintain  awareness  rather  than  as  a  specific  input  into  a  decision. 
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D.  Levels  of  Automation  in  Technology  Forecasting 

Most  Federal  efforts  to  forecast  new  and  emerging  technologies  and  applications  are 
completely  unautomated.  Automated  efforts  tend  to  be  focused  on  methods  that  extract 
infonnation  on  the  current  state  of  technology  from  vast  amounts  of  data,  but  do  not  have 
a  predictive  element.  In  this  section,  we  discuss  the  different  levels  and  approaches  that 
interview  respondents  said  they  are  taking  to  automate  forecasting. 

1.  Agencies  that  Use  Automated  Methods 

Only  two  of  the  agencies  at  which  we  conducted  interviews  currently  use  automated 
tools,  and  only  one  of  those  agencies  uses  automated  methods  for  forecasting  (though  it  is 
not  technology  forecasting).  The  EIA  prepares  forecasts  of  energy  demands  by  technology 
and  by  sector.  The  forecasts  are  based  on  the  National  Energy  Modeling  System  (NEMS), 
which  is  an  integrated  model  of  the  U.S.  energy  demands  developed  internally  at  EIA. 
NEMS  “projects  the  production,  imports,  conversion,  consumption,  and  prices  of  energy, 
subject  to  assumptions  on  macroeconomic  and  financial  factors,  world  energy  markets, 
resource  availability  and  costs,  behavioral  and  technological  choice  criteria,  cost  and 
perfonnance  characteristics  of  energy  technologies,  and  demographics”  (Energy 
Infonnation  Administration  2009).  NEMS  has  been  used  to  examine  the  impacts  of  existing 
or  potential  energy  policies,  as  well  as  new  energy  technologies.  To  develop  accurate 
forecasts,  NEMS  includes  models  for  how  technologies  improve  over  time.  For  certain 
energy  sectors,  EIA  considers  individual  technology  characteristics,  including  the  “initial 
capital  cost,  operating  cost,  date  of  availability,  [and]  efficiency”  (Energy  Infonnation 
Administration  2009).  To  model  this  data  well,  EIA’s  modelers  should  be  aware  of  the 
state  of  the  art  of  each  technology  and  its  likely  trajectory  for  improvement.  However,  this 
information  is  only  used  as  an  input  to  the  model;  the  model  itself  does  not  produce  a 
technology  forecasting. 

The  Directorate  of  Science  and  Technology  at  the  Central  Intelligence  Agency 
leverages  commercial  tools  for  data  analytics.  They  have  ongoing  funded  efforts  exploring 
emerging  analytics  technologies  for  forecasting.  Respondents  at  the  Office  of  the  Director 
of  National  Intelligence  Office  of  Acquisition  similarly  indicated  having  used  commercial 
tools  to  watch  new  and  emerging  technologies,  though  they  do  not  use  any  automated  tools 
for  technology  forecasting. 

2.  Agencies  Currently  Pursuing  Automated  Methods 

Most  of  the  agencies  that  are  currently  pursuing  automated  technology  forecasting 
methods  fund  scientific  research  and  make  mid-term  forecasts.  Currently,  automated  tool 
use  focuses  on  gap  analysis  and  understanding  the  current  state  of  the  field,  and  is  not 
predictive.  However,  there  is  interest  in  moving  towards  predictive  technologies. 
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At  NIB  IB,  technology  assessments  are  conducted  as  part  of  their  portfolio  analysis 
work.  The  goal  is  to  understand  and  address  funding  gaps.  The  gap  analysis  is  conducted 
using  IN-SPIRE  (discussed  in  more  detail  in  Appendix  C)  and  bibliometric  analysis  tools; 
once  potential  areas  are  identified,  expert  input  is  sought  to  detennine  research  areas  that 
NIB  IB  should  fund  in  the  future.  Interview  respondents  at  NIB  IB  expressed  interest  in  tools 
that  would  assist  in  other  aspects  of  portfolio  analysis,  such  as  peer  review,  allocation  of 
funds  and  understanding  how  collaborations  and  grants  come  together.  The  ultimate  goal 
is  to  be  able  to  make  better  decisions  about  what  research  should  be  funded. 

AFOSR  currently  uses  scientometric  analysis  and  large-scale  data  analysis  for 
portfolio  analysis  and  understanding  trends  in  basic  science  research  areas.  These  data 
analytics  tools  come,  in  part,  from  the  Basic  Research  Innovation  Collaboration  Center 
(BRICC),  with  which  AFOSR  is  collaborating.5  The  respondents  at  AFOSR  indicated  that 
they  do  not  currently  employ  any  approaches  for  predictive  analysis,  whether  automated 
or  manual.  However,  they  have  investigated  the  use  of  FUSE  to  identity  emerging  areas  of 
research,  and  previously  assessed  the  basic  areas  of  research  needed  to  support  the  long¬ 
term  technical  capabilities  needed  by  the  United  States  Air  Force.6 

DTRA  J9’s  goal  is  to  anticipate  emerging  threats  arising  from  technologies  associated 
with  chemical,  radiological,  biological,  and  nuclear  weapons  of  mass  destruction.  While 
the  respondents  at  DTRA  J9  would  like  to  use  a  predictive  tool  or  system,  currently  their 
effort  is  based  on  an  assessment  of  observable  trends,  which  are  reviewed  by  subject  matter 
experts  within  the  Countering  Weapons  of  Mass  Destruction  (CWMD)  community  of 
practice.  Respondents  at  DTRA  J9  indicated  that  their  current  methodology  is  based  on 
anecdotal  evidence  and  is  not  rigorous;  they  would  like  to  explore  the  use  of  an  automated 
tool  that  is  analytically  driven. 

OTI  within  the  Office  of  the  ASD(R&E)  leverages  commercial  technology 
forecasting  systems  and  invests  in  the  development  of  new  forecasting  approaches  that 
integrate  multiple  data  analysis  streams  and  a  suite  of  science  and  technology  forecasting 
tools  (including  a  technology  watch  and  a  horizon  scanning  tool,  which  are  discussed 
further  in  Appendix  C).  Currently,  the  OTI  tool  uses  data  analytics,  but  is  not  predictive. 
However,  the  goals  of  the  technology  watch  and  horizon  scanning  tool  involve  forecasting, 
since  they  are  to  “identify  emerging  S&T  areas  that  will  enhance  operational  capability  in 
the  next  10-20  years,  assess  scientific  areas  that  should  be  included  in  the  long-range  (10+ 
years)  research  strategy,  and  provide  context  to  understand  the  S&T  that  will  result  in 


5  BRICC  website,  “BRICC  Projects,”  http://bricc.vt-arc.org/. 

6  Forthcoming  report  from  the  IDA  Science  and  Technology  Policy  Institute. 
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significant  new  discoveries.”7  The  respondents  at  OTI  said  that  they  were  interested  in 
developing  or  using  a  predictive  forecasting  capability;  however,  they  view  their  role  as 
integrating  rather  than  developing  advanced  forecasting  tools,  and  they  would  need 
agencies  like  IARPA  to  develop  those  tools.  OTI  respondents  further  mentioned  that,  even 
with  an  automated  system,  they  envision  having  subject  matter  experts  involved  to  refine 
the  tool  output. 

3.  Agencies  Not  Pursuing  Automated  Processes 

Personnel  at  the  three  agencies  that  are  not  pursuing  automated  processes  each  focus 
on  short-term  technology  forecasting,  in  which  the  technologies  are  already  in  late 
development  stages,  and  the  role  of  the  office  is  principally  to  understand  the  potential 
implications  of  the  application  of  these  technologies  in  the  short  tenn.  These  agencies  also 
base  their  projections  on  a  wide  variety  of  inputs  in  addition  to  scientific  literature,  thereby 
capturing  not  only  the  technology  aspects  but  also  market  factors  and  economic  and 
national  security  impacts. 

Respondents  were  unfamiliar  with  publicly  available  automated  tools  and  seemed  to 
have  neither  the  time  nor  the  staffing  resources  to  pursue  new  methods.  Despite  their  not 
having  automated  tools,  their  technology  assessment  and  forecasting  processes  may  be 
complex. 

BIS  in  DOC  tracks  the  evolution  of  potential  dual-use  emerging  technologies  and 
determines  whether  their  export  should  be  regulated.  (This  is  discussed  in  more  detail  in 
Appendix  D.)  A  committee  of  technical  experts  identify  candidate  technologies  for  export 
control  regulation.  In  this  process,  a  factors  are  examined  that  range  from  the  stage  of 
technology  development,  to  the  impact  of  global  sales  of  the  technology,  to  the  effect  on 
industry  of  restricting  sales.  The  process  is  geared  towards  looking  at  technologies  in  late 
stages  of  development,  and  the  analysis  focuses  on  how  export  of  these  technologies  could 
trigger  a  situation  of  national  security  concern  in  the  future. 

GAO’s  technology  assessments  look  at  the  anticipated  trajectory  of  emerging 
technologies,  cost  factors,  and  economic  and  policy  implications.  The  manual  process 
relies  on  literature  reviews  by  librarians  and  consultations  with  subject  matter  experts.  The 
respondents  at  GAO  expressed  that  one  of  their  most  significant  challenges  is  forecasting 
the  global  economic  impact  of  technologies,  and  they  said  that  they  would  benefit  from  the 
use  of  a  tool  or  methodology  to  identify  historical  patterns  that  have  shaped  the  trajectory 
of  general  purpose  technologies,  and  apply  the  insights  to  their  studies. 


7  Department  of  Defense  Research  &  Engineering  Enterprise  website,  “Science  &  Technology  Corner, 
Technology  Watch  and  Horizon  Scanning  for  the  Department  of  Defense,” 
http://www.acq.osd.mil/chieftechnologist/cto/cto_TWHS.html. 
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AHRQ  within  the  Department  of  Health  and  Human  Services  has  the  most  structured 
non-automated  technology  forecasting  system  that  was  encountered  during  the  interviews. 
AHRQ  has  developed  a  horizon  scanning  system  called  the  Healthcare  Horizon  Scanning 
System,  a  10-step  process  for  identifying  emerging  technologies  and  healthcare 
innovations.  This  system  is  discussed  in  more  detail  as  a  case  study  in  Appendix  D.  In  the 
Healthcare  Horizon  Scanning  System,  medical  librarians  scan  publicly  available  sources 
such  as  medical  journals,  scientific  literature,  company  press  releases,  trade  publications, 
and  newsletters  for  leads.  Analysts  and  topical  experts  review  and  analyze  all  leads  in  a 
process  combining  validation  and  in-depth  research.  The  final  output  is  used  to  inform 
healthcare  companies  and  the  general  public  on  potentially  high-impact  healthcare 
products  and  trends.  This  method  involves  examining  the  future  implications  of  a 
healthcare  innovation  if  it  becomes  publicly  actionable,  rather  than  forecasting  future 
technologies. 

E.  Trust 

Trust  emerged  as  a  key  topic  during  interview  discussions  of  the  current  state  of 
technology  forecasting.  Responses  varied  from  the  extreme  trust  in  and  preference  for  an 
automated  tool,  with  respondents  viewing  such  tools  as  agnostic  to  human  prejudice,  to 
complete  distrust  of  anything  other  than  an  SME  who  is  familiar  to  the  respondent. 
According  to  respondents  both  within  and  outside  the  IC  and  DOD,  this  distrust  extends  to 
other  types  of  automated  tools,  not  just  those  used  for  forecasting. 

Almost  all  respondents  relied  heavily  on  SMEs  for  their  interpretations  of  the  future 
of  technology;  many  relied  entirely  on  SMEs.  Trust  in  human  judgement  over  data  or 
models  was  the  general  justification  for  this.  Many  respondents  also  reported  that  even  if 
technology  forecasts  were  used,  especially  computer-generated  models,  they  would,  at 
most,  be  used  to  verify  or  support  the  opinions  of  trusted  SMEs. 

Much  of  the  reliance  on  SMEs  appeared  to  be  the  result  of  the  role  of  analysts  within 
the  larger  Federal  Government  structure  and  the  interaction  between  analysts  and  their 
bosses — the  decision-maker.  Reasons  cited  for  including  the  opinions  of  SMEs  in  the 
information  chain  included  accountability  of  the  analyst  to  the  decision-maker,  potential 
distrust  of  the  decision-maker,  and  concern  about  what  might  be  done  with  information  if 
it  were  divorced  from  analyst-provided  context.  One  respondent  noted  that  computer 
models  cannot  be  questioned  or  held  accountable  for  bad  decisions  or  incorrect  forecasts 
in  the  same  way  that  a  human  could  be. 
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F.  Discussion  of  the  Current  State  of  Technology  Forecasting  in  the 
Federal  Government  and  its  Implications  for  Future  Forecasting 
Tools 

The  landscape  of  technology  forecasting  efforts  in  the  Federal  Government  shows  a 
substantial  amount  of  variation.  Agencies  are  interested  in  forecasting  timescales  ranging 
from  under  5  years  in  the  future  to  30  years  in  the  future.  They  are  also  in  different  stages 
in  tenns  of  their  pursuit  of  automation  for  forecasting.  Table  1  summarizes  the  timescale 
and  automation  bins  in  which  each  agency  belongs.  Note  that  some  agencies  belong  in 
multiple  bins,  or  they  currently  reside  in  one  bin,  but  have  the  goal  of  progressing  to  a 
different  bin. 

The  technology  forecasting  activities  of  most  agencies  are  specific  to  the  mission  of 
the  particular  agency.  This  is  true,  even  when  the  agency  is  interested  in  horizon  scanning 
activities.  The  “area  of  interest”  of  the  horizon  may  be  as  specific  as  a  single  program 
manager’s  portfolio.  Sometimes,  an  agency  does  not  get  to  define  with  technologies  it 
examines.  In  the  case  of  GAO  and  the  BIS  Office  of  Technology  Evaluation,  the 
technologies  to  be  examined  are  assigned  for  investigation.  In  these  cases,  in  which  a 
technology  is  in  late  stages  of  development,  the  focus  of  the  forecasting  effort  is  on  the 
downstream  implications  of  the  technology. 


Table  1 .  Agencies  Grouped  by  Their  Forecasting  Timescale,  and  Whether  They  Currently 
Are  Using,  Pursuing,  or  Not  Pursuing  Automated  Methods 


Forecasting  Timescale 

Short-term 

Mid-term 

Long-term 

(1-5  years) 

(5-20  years) 

(20-30  years) 

Using  Automated 
Methods 

CIA,  EIA* 

EIA* 

EIA* 

Pursuing 

Automated  Methods 

DTRA  J9  (currently) 

AFOSR,  DTRA  J9  (goal), 
DOD  OTI  (ASD[R&E]), 
NIBIB 

Neither  Using  nor 
Pursuing 

Automated  Methods 

AHRQ,  BIS,  GAO 

*  While  EIA  produces  forecasts  on  these  timescales,  they  are  not  technology  forecasts. 


Some  agencies  have  not  explored  automation  at  all,  even  when  commercial  or  free 
resources  are  available  to  streamline  their  process.  In  contrast,  EIA  uses  the  National 
Energy  Modeling  System  to  do  sophisticated  modeling  that  integrates  economic, 
technological,  and  demographic  data. 

At  multiple  agencies  that  make  decisions  based  on  anticipated  technology 
developments,  the  focus  is  on  improving  awareness  of  the  current  state  of  research  and  new 
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technology  developments  rather  than  future  technologies  or  applications.  This  seems  to  be 
due  to  a  combination  of  two  factors:  (1)  analysts  are  overwhelmed  by  the  volume  of 
information  that  they  need  to  be  aware  of,  (2)  technology  development  cycles  may  be  too 
fast  for  analysts  to  keep  up  with  the  technology. 

Tools  could  potentially  help  address  both  of  these  problems.  In  the  case  of  analysts 
being  overwhelmed  with  information,  tools  could  help  integrate,  summarize,  and  highlight 
key  pieces  of  information.  In  the  case  of  fast  technology  development  cycles,  tools  could 
be  used  to  constantly  scan  the  horizon  for  indicators  to  keep  analysts  abreast  of  new 
developments. 

Given  both  the  decision-making  structure  in  the  government  and  the  significant  role 
of  analysts  in  contextualizing  information  for  decision-making,  it  is  likely  that  any 
technology  forecasting  tools  or  their  outputs  produced  in  the  near-term  would  be  used  by 
analysts  or  program  managers,  and  not  by  higher  level  decision-makers.  The  managerial 
structure  also  means  that  if  a  technology  forecast  is  used  by  analysts  or  program  managers, 
the  forecast  would  likely  be  one  of  multiple  inputs  into  a  decision,  rather  than  a  single  input 
to  a  decision. 

In  general,  respondents  reported  a  willingness  to  use  automated  tools  but  only  with  a 
high  level  of  data  verification  and  an  understanding  of  data  inputs.  Respondents 
overwhelmingly  felt  that  the  role  of  the  human  in  the  loop  was  critical  to  the  use  of 
forecasting  tools  in  the  decision-making  process.  An  output  from  a  tool,  according  to 
respondents,  would  most  likely  then  be  further  interpreted,  contextualized,  and  explained 
by  analysts  prior  to  being  elevated  to  the  level  of  a  decision-maker.  Therefore,  any  tool 
should  be  designed  to  aid  rather  than  to  supplant  the  role  of  the  analyst,  and  it  should  not 
be  operated  in  a  stand-alone  fashion.  In  the  following  chapter,  the  specific  capabilities  of 
automated  tools  that  could  provide  support  to  analysts  are  discussed. 
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3.  Desired  Capabilities  of  Automated  Tools 


During  interviews,  respondents  identified  specific  capabilities  that  could  help  them 
with  their  forecasting  work  as  well  as  general  needs  that  could  potentially  be  addressed  by 
automated  tools.  Some  of  the  automated  tools  that  could  help  would  not  necessarily  be 
considered  technology  forecasting  tools,  since  they  would  not  be  predictive.  This  chapter 
distills  the  different  capabilities  that  could  be  useful  to  the  personnel  interviewed  and  to 
others  in  the  Federal  Government  with  similar  needs.  The  capabilities  are  divided  into  three 
categories:  (1)  forecasting  capabilities,  which  provide  qualitative  information  about  a 
future  state,  (2)  alert  capabilities,  which  would  flag  a  user  if  a  pre-determined  potential 
future  state  is  likely  to  occur,  and  (3)  tracking  and  summarization  capabilities,  which  give 
the  user  information  about  the  current  state,  contextualized  by  the  past.  The  goals  of  each 
of  these  capabilities  and  the  relationships  to  a  potential  future  state  are  shown  in  Table  3. 


Table  3.  Categories  of  Capabilities  Desired  by  Interview  Respondents 


Capability 

Goal 

Relationship  to  a 
Potential  Future  State 

Forecasting 

Provide  qualitative 
information  about  a  future 
state 

Predicts  a  future  state 

Alert 

Notify  users  that  a  pre¬ 
determined  potential  future 
state  is  likely  to  occur  (with  a 
pre-specified  level  of 
probability) 

Pre-determined  indicators 
relate  to  a  specific  future 
state 

Tracking  and  Summarization 

Provide  information  about  the 
current  state 

Provides  no  information 
about  any  potential  future 
state 

While  multiple  agencies  may  find  one  or  more  of  the  capabilities  discussed  here 
useful,  the  application  of  that  capability  would  likely  depend  on  each  agency’s  mission. 
Consequently,  the  “user  requirements”  for  a  given  capability  at  different  agencies  would 
likely  vary. 


A.  Forecasting  Capabilities 

Relatively  few  interview  respondents  expressed  interest  in  a  capability  that  would 
provide  the  user  with  qualitative  information  about  a  future  state.  The  respondents  that  did 
express  an  interest  would  be  interested  in  capabilities  that  identify  trends  in  current 
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technology  and  provide  a  basis  for  predicting  the  future  characteristics  of  a  technology  and 
the  future  applications  of  a  technology.  These  capabilities  are  discussed  in  more  detail 
below. 

1.  Extrapolation  and  Trend  Analysis 

In  extrapolation  and  trend  analysis,  a  user  or  tool  uses  information  from  the  past  and 
present  to  forecast  a  future  state.  These  techniques  assume,  according  to  the  National 
Research  Council  (NRC)  study,  Persistent  Forecasting  of  Disruptive  Technology,  that  “the 
future  represents  a  logical  extension  of  the  past  and  that  predictions  can  be  made  by 
identifying  and  extrapolating  the  appropriate  trends  from  the  available  data”  (NRC  2010). 
This  suggests  that  trend  extrapolation  and  analysis  work  can  work  well  if  the  historical 
forces  that  drive  the  trend  remain  the  same,  but  can  fail  if  those  forces  change  (NRC  2010). 
The  NRC  study  presents  four  different  types  of  extrapolation  and  trend  analysis:  (1)  trend 
extrapolation,  in  which  recent  technology  trends  are  projected  into  the  future;  (2)  Gompertz 
and  Fisher-Pry  substitution  analysis,  in  which  a  technology’s  evolution  is  assumed  to  be 
an  “S-shaped”  growth  curve  as  it  is  developed,  deployed,  and  reaches  maturity;  (3) 
analogies,  in  which  information  about  specific  previous  technologies  and  situations  are 
used  to  examine  the  evolution  of  a  current  technology;  and  (4)  morphological  analysis, 
which  is  a  structured  process  that  applies  general  infonnation  about  technology  evolution 
to  examine  how  a  new  technology  could  evolve. 

Trend  extrapolation  was  mentioned  as  a  capability  of  interest  by  interview 
respondents  at  DTRA  J9.  The  appeal  of  trends  is  that  they  are  closely  tied  to  observable 
infonnation  and  they  could  be  presented  to  analysts  for  interpretation.  Respondents  at  GAO 
also  expressed  interest  in  trend  extrapolation,  as  well  as  in  analogies.  The  respondents  said 
that  they  would  like  to  have  a  tool  that  can  identify  historical  patterns  for  general  purpose 
technologies  and  apply  those  trends  and  patterns  to  assess  new  technologies  to  examine  the 
direction  that  they  will  take.  Morphological  analysis,  which  uses  principles  of  technology 
evolution  rather  than  specific  examples  of  relevant  technology  evolution,  could  also 
potentially  be  useful  to  GAO,  since  the  respondents  expressed  interest  in  understanding  the 
factors  that  enable  technology  development.  While  not  mentioned  by  EIA  respondents, 
substitution  analysis  could  potentially  be  helpful  with  predicting  the  costs  and  diffusion  of 
specific  energy  technologies  in  the  NEMS  models. 

Extrapolation  and  trend  analysis  tools  would  likely  be  most  useful  for  agencies  that 
are  focused  on  forecasting  technologies  in  middle  to  late  development  stages.  For  agencies 
attempting  to  make  forecasts  about  technologies  or  applications  20-30  years  in  the  future, 
there  may  not  be  enough  data  to  establish  a  trend  since  the  technology  would  only  be  in 
early  development  stages,  with  little  evidence  of  its  development  pathway.  Similarly,  for 
agencies  that  focus  on  technologies  with  fast  development  cycles,  there  may  not  be  enough 
time  to  establish  a  clear  trend  before  the  technology  reaches  maturity. 
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2.  Dual-Use  Probability  Calculation 

A  variation  on  the  ability  to  forecast  the  future  state  of  a  technology  by  extrapolating 
historical  trend  data  is  the  ability  to  project  possible  trajectories  for  a  technology  given 
varying  sets  of  driving  forces.  One  of  the  particular  applications  of  interest  would  be  a  tool 
that  is  able  to  determine  the  probability  of  a  technology  at  an  early  development  stage 
becoming  a  dual-use  technology. 

At  early  technology  development  stages,  the  different  trajectories  that  a  given 
technology  could  take  may  result  in  a  range  of  potential  future  characteristics  and 
applications.  While  predicting  the  exact  form  of  applications  that  will  dominate  in  the 
future  is  difficult  at  best,  it  is  possible  to  make  a  “range  of  forecasts”  of  performance 
characteristics  and  applications  in  the  future  by  considering  the  factors  that  shape  the 
development  and  adoption  of  a  given  technology  in  its  different  applications.  By 
determining  which  final  states  have  military  and  civil  applications,  and  the  likelihood  of 
these  states  occurring,  it  could  be  possible  to  detennine  the  likelihood  that  a  technology 
becomes  dual  use. 

A  dual-use  capability  could  be  used  by  DTRA  J9  and  BIS  in  DOC.  DTRA  J9’s  goal 
is  to  anticipate  emerging  threats  related  to  weapons  of  mass  destruction  and  provide  timely 
responses.  While  DTRA  J9  is  made  aware  of  recent  and  emerging  threats  (corresponding 
to  technologies  in  late  development  stages),  if  they  were  able  to  detennine  the  likelihood 
that  an  early  stage  technology  could  become  a  threat,  they  would  have  greater  lead  time  to 
respond.  Currently,  DTRA  J9  manually  collects  infonnation  on  trends,  assesses  the  trends, 
and  has  the  assessments  analyzed  by  DTRA  and  the  CWMD  community  of  practice  to 
identify  implications  for  weapons  of  mass  destruction.  This  process  is  seen  by  the 
respondents  as  lacking  analytical  rigor  and  largely  based  on  anecdotal  knowledge  and 
evidence;  the  respondents  at  DTRA  J9  therefore  expressed  interest  in  institutionalizing  a 
technology  forecasting  capability  aimed  at  identifying  emerging  threats.  While  the  office 
is  currently  exploring  automated  trend  analysis,  a  tool  that  is  able  to  calculate  a  dual-use 
threat  could  be  added  to  this  capability  to  select  which  technologies  should  be  further 
investigated  by  analysts.  The  BIS  OTE,  which  assesses  new  technologies  and  their 
potential  for  dual-use  applications,  could  also  benefit  from  such  a  tool.  Currently,  potential 
dual-use  technologies  are  selected  by  external  groups,  such  as  the  Emerging  Technology 
and  Research  Advisory  Committee,  and  then  given  to  OTE  to  conduct  in-depth  analysis. 
A  dual-use  calculation  tool  could  help  ensure  that  OTE  is  examining  the  technologies  with 
high  probabilities  of  becoming  dual  use.  Such  a  tool  would  also  be  helpful  throughout  the 
IC  and  DOD. 

3.  Prediction  Markets 

In  prediction  markets,  the  collective  judgement  of  a  group  of  individuals  is  used  to 
generate  forecasts  about  an  event  or  parameter.  Predictions  are  treated  “as  assets  to  be 
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traded  on  a  virtual  market  that  can  be  accessed  by  a  number  of  individuals....  The  final 
market  value  of  the  asset  is  taken  to  be  indicative  of  its  likelihood  of  occurring”  (NRC 
2010).  As  an  example,  to  understand  the  probability  of  a  candidate  winning  an  election, 
participants  would  say  whether  or  not  they  think  that  the  candidate  will  win  the  election. 
Incentives  can  be  set  up  such  that  the  participant  must  pay  to  make  a  prediction  and  receives 
a  payout  depending  on  the  outcome  of  the  event  (Wolfers  and  Zitzewitz  2004),  or  such  that 
participants  are  rewarded  for  participation  or  accuracy  (George  Mason  University  2015). 
The  principle  behind  using  a  prediction  market  approach  is  that  the  aggregation  of  a  set  of 
judgements  generally  generates  more  accurate  forecasts  than  individual  experts  (George 
Mason  University  2015).  SciCast,  a  prediction  market  project  funded  by  IARPA  and 
developed  by  George  Mason  University,  is  briefly  summarized  in  Appendix  C. 

Two  respondents,  one  from  AHRQ  and  one  from  NIBIB,  expressed  interest  in 
crowdsourced  tools  related  to  prediction  markets.  Currently,  AHRQ  has  a  non- automated 
operational  forecast  system  called  the  Healthcare  Horizon  Scanning  System  to  identify 
innovations  that  could  meet  previously  unmet  health  needs.  (This  system  is  discussed  in 
more  detail  in  Appendix  D.)  A  crowdsourced  prediction  system  could  allow  participants  to 
suggest  additional  innovations,  as  well  as  to  make  predictions  related  to  innovations. 
According  to  the  respondent,  NIBIB  has  explored  posing  research  challenges  to  the  general 
public  and  receiving  feedback,  and  the  respondent  could  envision  expanding  the 
crowdsourcing  efforts  further. 

Prediction  markets  could  be  used  to  help  agencies  with  a  wide  range  of  forecasting 
needs,  though,  to  maintain  interest  from  participants,  it  would  likely  be  best  used  for  short¬ 
term  forecasts,  which  could  be  proven  or  disproven  in  a  relatively  short  timescale. 
Otherwise,  the  incentives  would  need  to  be  for  participation  rather  than  accuracy,  since  the 
potential  payoff  would  occur  too  far  in  the  future.  Within  short-term  forecasting  needs, 
prediction  markets  would  be  well  suited  for  fast-moving  technology  development  cycles, 
which  have  few  indicators  that  could  be  tracked.  In  these  cases,  properly  developed 
questions  and  incentives  could  be  used  to  get  experts  with  inside  knowledge  to  participate, 
giving  rise  to  knowledge  that  would  not  be  otherwise  accessible  to  Federal  experts. 

B.  Alert  Capabilities 

A  respondent  from  BIS’s  Office  of  Technology  Evaluation  suggested  that  an  alert 
tool  that  raises  “red  flags”  when  a  pre-detennined  threshold  is  crossed  could  be  developed 
to  alert  an  analyst  to  the  need  to  conduct  an  in-depth  analysis.  In  general,  such  a  tool  would 
involve  having  analysts  first  detennine  what  future  states  they  are  concerned  about.  Then, 
they  could  determine  what  indicators  would  give  infonnation  about  the  likelihood  of  those 
future  states  occurring,  and  the  thresholds  for  those  indicators  at  which  the  analyst  wants 
to  be  alerted.  These  tools  are  distinct  from  forecasting  tools  because  they  do  not  give  any 
information  about  the  future;  they  simply  report  on  infonnation  from  the  past  and  present. 
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However,  the  red  flag  implies  a  future,  because  the  analyst,  by  determining  the  relevant 
information  inputs  and  thresholds,  sets  up  a  relationship  between  the  information  passing 
a  threshold  and  a  potential  future  state. 

This  class  of  red  flag  capabilities  could  support  a  range  of  analysts  by  allowing  the 
user  to  specify  the  type  of  input  data  and  threshold.  In  basic  research  applications,  a 
program  manager  might  want  the  tool  to  draw  from  certain  scientific  journals  or  conference 
proceedings,  and  the  thresholds  could  be  counts  of  key  words  or  authors  appearing,  or  the 
co-occurrence  of  key  words  and  authors.  Analysts  that  look  at  technologies  in  later  stages 
of  development  might  have  interest  in  the  tool  drawing  from  patent  applications,  or  the 
appearance  of  key  words  in  trade  journals.  Because  of  the  range  of  customization  possible, 
this  class  of  tool  could  potentially  be  used  to  support  short-,  medium-,  and  long-tenn 
forecasting  efforts. 

C.  Tracking  and  Summarization  Capabilities 

Many  interview  respondents  discussed  a  desire  to  be  more  aware  of  all  of  the 
information  relevant  to  their  fields,  without  specifying  any  desire  for  predictive 
technologies.  While  this  task  may  be  seen  as  simpler,  several  capabilities  that  are  currently 
desired  by  the  interview  respondents  have  not  yet  been  realized.  In  some  cases,  the  agencies 
seem  to  be  unaware  or  unable  to  use  existing  technologies.  (This  lack  or  ability  is  discussed 
in  more  detail  in  Chapter  2,  Section  D,  and  existing  tools  and  technologies  are  discussed  in 
Appendix  C.)  In  other  cases,  there  is  fundamental  research  needed  to  develop  a  new 
capability  or  tool.  These  needs  are  discussed  in  the  subsections  that  follow. 

1.  Tracking  Technology  through  Name  Changes 

As  technologies  move  through  the  maturation  cycle  from  the  research  phase  into 
applications  and  product  development,  the  same  product  or  entity  may  be  referred  to  by 
different  names  through  the  process.  Likewise,  data  sources  reporting  the  technology  and 
the  tenninology  and  language  associated  with  it  also  change.  For  example,  it  is  often  the 
case  that,  as  a  technology  matures  and  products  move  from  research  to  commercialization 
phase,  one  set  of  terminology  diminishes  and  a  different  set  of  tenninology  emerges  in  the 
literature.  One  of  the  challenges  associated  with  tracking  technologies  over  a  long  term  is 
to  be  cognizant  of  the  name  and  terminology  changes  associated  with  it  over  the  life  cycle 
of  its  development.  Techniques  to  automate  tracking  of  this  information  would  be  very 
useful. 

2.  Identifying  and  Tracking  Critical  People  in  a  Field  of  Research 

The  issue  of  detennining  critical  people  in  a  field  of  research  was  brought  up  in 
multiple  interviews.  Respondents  at  AFOSR  consider  one  of  their  chief  roles  to  be 
investing  in  people.  They  expressed  interest  in  tools  to  identify  leading  scientists  in  a  new 
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or  emerging  area  of  research,  as  a  way  to  not  only  understand  the  state  of  leading  edge 
research  in  the  field  but  also  identify  promising  people  to  invest  in.  In  addition,  they  would 
like  to  be  able  to  identify  which  young  investigators  have  the  most  promise.  A  similar 
interest  was  expressed  by  the  respondent  at  the  NIB  IB,  who  said  that  adding  a  capability 
to  identify  critical  people  in  the  biomedical  research  networks  would  strengthen  current 
efforts  to  manage  research  portfolios.  Tools  that  have  the  capability  to  provide  a  people 
forecast  by  tracing  how  people  coalesce  and  how  fields  grow  as  a  result  could  also  be  of 
interest  to  program  managers  in  various  agencies. 

The  indicators  of  a  researcher’s  success  in  innovating  and  commercializing  research 
technology  in  the  field  of  biotechnology  have  been  studied  by  Lynne  Zucker  and  Michael 
Darby,  and  could  be  used  to  inform  the  development  of  new  tools.  Zucker  and  Darby  refer 
to  biotechnology  researchers  that  lead  both  scientific  innovations  and  commercialization 
of  technologies  stemming  from  the  research  as  “star  scientists.”  They  contend  that 
knowledge,  particularly  when  it  is  new,  is  embodied  in  particular  individuals  and  cannot 
diffuse  rapidly  (Zucker  and  Darby  1998).  They  have  also  showed  that  close  collaborations 
between  star  scientists  in  academia  and  industrial  scientists  were  needed  to  accomplish 
commercialization  of  the  breakthroughs,  and  that  where  and  when  star  scientists  were 
actively  producing  publications  were  key  predictors  of  where  and  when  commercial  firms 
began  to  use  biotechnology  (Zucker  and  Darby  1996).  A  social  network  map  of  researchers 
in  which  collaborations  between  academic  and  industrial  scientists  are  highlighted  could 
be  used  to  help  analysts  identify  researchers  and  innovations  to  track. 

3.  Organizing  and  Visualization  Information 

In  many  interviews,  respondents  indicated  that  a  tool  that  organized  and  displayed 
information  would  be  extremely  useful,  especially  if  the  tool  had  intuitive  visualization 
capabilities.  Tools  that  could  potentially  be  used  have  not  been  adopted  by  the  respondents 
interviewed  for  this  project.  (Two  such  tools  are  Sci2,  which  is  an  open  source  tool,  and 
IN-SPIRE,  developed  by  Pacific  Northwest  National  Laboratory,  both  of  which  are 
discussed  in  Appendix  C.)  This  suggests  that  (1)  respondents  are  unaware  of  existing  tools; 

(2)  they  do  not  have  the  time  or  financial  or  personnel  resources  to  acquire  the  tools;  or 

(3)  the  tools  do  not  fully  meet  their  needs.  Discussions  indicate  that,  to  some  extent,  the 
current  tools  do  not  fully  meet  their  needs.  A  fully  developed  organization  and  visualization 
tool  that  would  be  useful  to  agency  personnel  would  be  one  that  allows  users  to  select  data 
sources  of  interest  and  suggests  additional,  complementary  data  sources  to  supplement  the 
user-selected  data  sources.  The  tool  would  continuously  receive  updates  from  those  data 
sources.  The  tool  could  then  have  a  number  of  different  analysis  capabilities  such  as:  (1) 
summarizing  the  information  within  a  single  data  source  or  across  data  sources  by  topic 
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modeling  or  generating  word  clouds;8  (2)  trend  analysis,  which  would  allow  the  user  to 
look  at  the  change  in  usage  of  a  tenn  or  topic  over  time;  (3)  highlight  new  terms  or  topics; 
and  (4)  allow  the  user  to  examine  co-occurrence  (this  could  be  applied  to  words  in  a 
document,  authors  of  publications,  etc.).  If  such  a  tool  were  made  flexible  enough  to 
accommodate  different  data  sources,  analysis  types,  and  user-specified  queries,  it  would 
be  highly  desirable  for  analysts  across  a  wide  range  of  missions. 

D.  Discussion  of  the  Desired  Capabilities  of  Automated  Tools 

While  there  does  not  appear  to  be  a  single  tool  or  capability  that  is  likely  to  meet  all 
requirements,  there  is  definite  interest  in  forecasting,  alert,  and  tracking  and  summarization 
capabilities  that  could  be  tailored  to  specific  goals  or  missions.  While  the  specific 
applications  of  the  capabilities  would  vary,  some  common  desired  characteristics  emerged, 
based  on  the  usage  model  of  the  tools  and  the  culture  of  trust  and  acceptance  of  automated 
tools  in  providing  input  to  the  decision-making  process.  The  common  desired  capabilities 
for  tools  are  (1)  that  they  present  information  in  an  easy-to-visualize  way  (discussed 
above);  (2)  that  they  are  customizable  to  a  user’s  specific  needs;  and  (3)  that  they  are 
trustworthy. 

A  tool’s  customizability  is  important  because  of  analysts’  different  forecasting 
needs.  For  instance,  the  desired  timescale  of  forecasts  varied  from  near  term 
(approximately  1  year)  to  long  term  (approximately  30  years).  Additionally,  the  level  of 
certainty  desired  of  forecasts  varied  significantly.  In  general,  the  level  of  desired  accuracy 
and  precision  changes  based  on  the  timescale  of  the  forecast  (i.e.,  near-tenn  forecasts  need 
to  be  more  accurate  and  precise  than  long-tenn  forecasts).9 

The  trustworthiness  of  a  tool  is  important  because,  as  described  by  multiple 
respondents,  analysts  put  their  credibility  on  the  line  when  they  bring  infonnation  to 
decision  makers.  The  analyst  needs  to  be  comfortable  with  the  tool’s  methodologies. 
Therefore,  an  important  requirement  for  any  tool  is  that  it  must  not  be  a  black  box,  but  has 
an  audit  trail  that  allows  the  methodology  (for  forecasting  or  other  tool  capabilities)  to  be 
transparent 


Word  clouds  show  the  popularity  of  different  words  in  a  single  document  or  set  of  documents  by 
creating  a  visualization  of  the  words  used  most  frequently,  in  which  the  font  size  of  the  word  correlates 
with  the  number  of  times  that  word  is  used. 

9  While  users  would  like  forecasts  with  high  levels  of  certainty,  users  do  not  currently  have  a  way  to 
quantify  their  current  accuracy,  precision,  or  recall;  therefore,  they  do  not  have  a  sense  of  what  these 
numbers  should  be  and  do  not  feel  strongly  about  the  level  at  which  a  tool  should  perform. 
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4.  Discussion  and  Implications  for  IARPA’s 
Technology  Forecasting  Efforts 


One  of  IARPA’s  roles  is  to  transition  research  results  to  agency  partners  in  the  IC. 
While  one  of  the  goals  of  this  project  was  to  help  inform  IARPA’s  development  of 
automated  technology  forecasting  tools,  the  interviews  revealed  many  obstacles  that 
IARPA  will  face  in  trying  to  develop  tools  that  will  transition  from  a  research  phase  to 
operational  use.  In  this  chapter,  we  discuss  these  obstacles  and  present  recommendations 
for  IARPA. 

A.  Current  State  of  Technology  Forecasting 

We  found  a  significant  amount  of  variation  in  forecasting  activities  within  the  Federal 
Government  in  tenns  of  the  timescales  of  interest,  the  level  of  automation  currently  used 
or  being  pursued,  and  the  level  of  trust  in  automated  methods  for  decision-making.  Few  of 
the  interview  respondents  currently  use  automated  tools  of  any  kind,  whether  or  not  they 
have  a  technology  forecasting  application.  Barriers  to  adoption  of  tools  include:  available 
tools  not  suiting  agency  needs,  comfort  with  the  current  process,  lack  of  time  to  train 
personnel  to  use  tools,  and  prohibitively  high  tool  costs.  In  addition,  interview  respondents 
cited  a  lack  of  trust  of  automated  methods  both  for  analysts  and  for  decision-makers. 

While  the  interview  respondents  are  interested  in  gaining  infonnation  about  future 
technology  characteristics  and  applications  for  short-,  mid-,  and  long-term  timescales, 
multiple  agencies  for  whom  part  of  the  mission  is  to  be  aware  of  emerging  technologies 
are  focusing  on  currently  available  new  technologies  and  applications  rather  than  future 
technologies  or  applications.  This  seems  to  be  due  to  a  combination  of  two  factors: 

(1)  analysts  are  overwhelmed  by  the  volume  of  infonnation  they  need  to  be  aware  of  and 

(2)  technology  development  cycles  may  be  too  fast  for  analysts  to  keep  up  with  the 
technology.  Both  of  these  factors  are  linked  to  the  general  lack  of  tool  use  in  the  Federal 
Government.  Increased  use  of  tools  could  help  analysts  maintain  awareness  about  new 
developments  and  organize  the  vast  amount  of  available  information. 

B.  Recommended  Capabilities  to  Develop  with  IARPA  Programs 

During  interviews,  respondents  identified  specific  capabilities  that  could  help  them 
with  their  forecasting  work,  as  well  as  general  needs  that  could  potentially  be  addressed  by 
automated  tools.  The  automatable  capabilities  can  be  divided  into  three  categories: 
(1)  tracking  and  summarization  capabilities,  which  give  the  user  information  about  the 
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current  state,  contextualized  by  the  past;  (2)  alert  capabilities,  which  would  flag  a  user  if  a 
pre-determined  potential  future  state  is  likely  to  occur,  and  (3)  forecasting  capabilities, 
which  provide  qualitative  infonnation  about  a  future  state. 

Tracking  and  summarization  capabilities  were  the  most  commonly  mentioned,  with 
many  respondents  expressing  a  desire  for  tools  to  assist  with  information  summarization 
and  situational  awareness.  These  tools  could  summarize  the  state  of  a  research  field,  a 
technology,  or  a  network  of  people.  One  possible  reason  for  interest  in  summarization  tools 
over  forecasts  could  be  an  issue  of  prioritization.  The  respondents  may  feel  that  they  are 
inadequate  at  “keeping  up”  with  the  current  state  of  their  field,  and  that  decisions  are  too 
often  made  on  the  basis  of  incomplete  or  anecdotal  information  simply  because  it  is  beyond 
the  capability  of  any  analyst  (or  decision-maker)  to  be  fully  updated  with  all  the 
information  that  should  ideally  provide  input  to  the  decision-making  process.  Decisions 
are  made,  regardless  of  whether  the  decision-maker  has  complete  infonnation.  An 
automated  approach  for  information  summarization  would  provide  a  more  comprehensive 
basis  for  decisions  than  decision-makers  currently  have. 

Alert  capabilities  could  flag  analysts  when  an  indicator  passes  a  pre-determined 
threshold,  notifying  them  to  focus  their  investigation  on  a  particular  issue.  While  not  a 
forecasting  tool,  alert  capabilities  could  be  used  to  inform  analysts  of  what  needs  to  be 
investigated  further,  which  could  help  them  understand  potential  future  threats  and 
opportunities. 

Within  the  desired  forecasting  capabilities,  interview  responses  mapped  onto  two 
established  techniques:  extrapolation  and  trend  analysis,  and  prediction  markets. 
Extrapolation  and  trend  analysis  tools  would  likely  be  most  useful  for  agencies  that  are 
focused  on  forecasting  technologies  in  middle  to  late  development  stages,  since  trend 
analyses  work  best  when  they  have  a  substantial  time  series  of  data  to  draw  from.  Prediction 
markets  would  be  best  used  for  short-term  forecasts,  which  could  be  proven  or  disproven 
in  a  relatively  short  timescale.  In  addition,  the  number  of  interview  respondents  that 
discussed  the  need  to  understand  whether  a  new  technology  is  likely  to  have  military 
applications  suggests  the  creation  of  a  new  tool,  one  that  would  calculate  the  probability 
that  a  technology  will  become  dual-use,  or  have  both  civilian  and  military  applications. 

C.  Considerations  in  the  Development  of  User-Oriented  Tools  for 
Technology  Forecasting 

For  a  tool  to  be  accepted  by  a  user,  it  must  (1)  present  infonnation  in  an  easy-to- 
visualize  way;  (2)  be  customizable  to  a  user’s  specific  needs;  and  (3)  be  trustworthy.  Each 
of  these  characteristics  is  discussed  in  more  detail  in  the  subsections  that  follow. 
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1.  Visualization 

A  tool  that  organizes  and  displays  information  in  an  intuitive  way  would  help 
analysts,  regardless  of  whether  the  tool  has  any  predictive  component.  Visualization  is  a 
key  component  for  tracking  and  summarization  tools,  as  well  as  for  extrapolation  and  trend 
analysis  tools,  which  would  allow  the  user  to  extract  key  infonnation  from  a  body  of  data 
inputs. 

2.  Customization 

One  challenge  that  IARPA  will  likely  face  in  developing  tools  for  use  in  the  Federal 
Government  is  the  high  degree  of  specificity  desired  by  potential  users,  which  may  make 
it  difficult  to  develop  an  “all  purpose”  tool.  Even  when  users  are  interested  in  broad 
technology  horizon  scanning  activities,  they  are  examining  technologies  within  the  context 
of  their  mission  and  goals.  Given  the  variety  of  use  cases  that  the  interviewed  personnel 
described,  a  single  tool  or  capability  is  not  likely  to  meet  all  the  requirements. 

3.  Trustworthiness 

Mistrust  of  automated  tools  is  likely  to  be  a  significant  barrier  to  transitioning  a 
technology  forecast.  This  lack  of  trust  is  reflected  in  the  desire  for  transparency  in  a 
technology  forecasting  tool.  Factors  that  are  non-intuitive  but  strong  predictors  would 
likely  have  a  low  rate  of  acceptance  by  users.  This  also  limits  the  desired  complexity  of 
technology  forecasting  tools  by  analysts.  While  respondents  did  not  acknowledge  that  they 
had  this  concern,  they  could  be  concerned  that  an  automated  forecast  could  supplant  their 
judgement  in  a  way  that  subject  matter  experts  do  not  currently  do.  As  researchers  and 
analysts,  their  job  is  to  gather  infonnation  and  use  their  expert  judgement  to  detennine  the 
future  implications  of  that  infonnation.  If  a  technology  forecast  detennines  future 
implications  automatically,  the  respondents  may  be  concerned  that  the  forecasts  would 
reduce  the  need  for  human  analysis. 

D.  Conclusions 

In  the  short  tenn,  for  a  technology  forecast  to  be  accepted  by  a  user,  it  needs  to  fit  into 
the  cunent  workflow  and  decision-making  structure.  Right  now,  automated  technology 
forecasting  tools  are  not  used  directly  for  decision-making.  This  may  be  part  of  the  desire 
for  summaries  compared  to  forecasts,  since  respondents  encounter  summaries  in  their  job 
currently,  so  they  are  more  familiar  with  them,  and  know  how  better  summaries  would  fit 
into  and  improve  their  current  workflow.  A  red-flag  or  alert  system  could  easily  be 
integrated  into  the  current  workflow,  since  it  would  run  in  the  background,  and  prompt  the 
analyst  to  investigate  certain  topics  further.  However,  automated  trend  analysis  or  crowd¬ 
sourcing  tools  are  not  widely  used  and  therefore  analysts  would  need  to  adjust  to  using 
these  tools  in  their  current  processes. 
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Respondents  overwhelmingly  expressed  that  the  role  of  the  human  in  the  loop  was 
critical  to  the  use  of  forecasting  tools  in  the  decision-making  process.  An  output  from  a 
tool,  according  to  respondents,  would  most  likely  then  be  further  interpreted, 
contextualized,  and  explained  by  analysts  prior  to  being  elevated  to  the  level  of  a  decision¬ 
maker.  Therefore,  any  tool  should  be  designed  so  as  to  aid  rather  than  to  supplant  the  role 
of  the  analyst,  and  should  not  be  operated  in  a  stand-alone  fashion. 
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Appendix  A. 

Interview  Methodology  and  Guide 


Sampling 

Because  there  is  little  information  currently  available  on  who  uses  technology  forecasting 
and  technology  forecasting  tools  across  the  Federal  Government,  random  sampling  or  other  more 
systematic  sampling  was  not  possible.  Instead,  a  primarily  respondent-driven  sampling  method 
was  used  to  identify  and  facilitate  contact  with  appropriate  individuals.  An  initial  list  of 
respondents  was  drawn  from  the  Technology  Watch  and  Horizon  Scanning  Community  of 
Practice.  Additional  respondent  information  or  candidate  government  offices  were  provided  by 
original  respondents.  Because  of  the  limitations  of  this  sampling  methodology  this  report  does  not 
claim  to  cover  all  users  of  technology  forecasting  within  the  Federal  Government.  Every  effort 
was  made,  however,  to  cover  a  broad  range  of  agencies  in  the  IC  and  DOD,  as  well  as  elsewhere. 

Interview  Guide  Development  and  Interview  Procedure 

Subject  matter  experts  from  both  within  and  outside  the  government  were  interviewed  for 
background  material  and  context  in  the  development  of  the  questions  for  the  semi-structured 
interview.  Initially,  two  interview  guides  were  developed:  one  for  primary  users  of  technology 
forecasting  tools  and  another  for  high-level  end-users  of  forecasts  and  high-level  decision  makers. 
However,  it  became  clear  during  our  initial  interviews  there  were  not  many  primary  tool  users  and 
that  the  distinction  between  forecasters  and  decision  makers  was  not  as  distinct  as  initially  thought. 
Because  of  this,  only  the  interview  guide  for  high-level  end-users  was  used  for  each  interview  and 
additional,  specific  questions  were  added  based  for  the  specific  role  of  the  respondent. 

Each  interview  was  semi-structured  and  based  on  the  same  interview  guide.  All  interviews 
were  conducted  by  the  same  core  interview  team.  Additional  interview  questions  were  developed 
specifically  based  on  the  role  of  the  respondent  and  the  mission  of  their  office  or  organization. 
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General  Interview  Guide:  Use  of  Technology  Forecasting  Tools  in 
Decision-Making — Federal  Government  Personnel 

Screening  questions: 

•  Do  you  use  technology  forecasts? 

•  Are  you  familiar  with  currently  available  technology  forecasting  tools? 

•  Do  you  have  a  need  for  technology  forecasts  and/or  technology  forecasting  tools?) 

Introduction  of  Personnel 

Introduction  and  Informed  Consent 

The  Office  of  the  Director,  National  Intelligence,  Intelligence  Advanced  Research  Projects 
Activity  (IARPA)  asked  the  Science  and  Technology  Policy  Institute  to  provide  information  on 
the  ways  that  technology  forecasting  and  technology  forecasting  tools  are  included  in  the  decision¬ 
making  process  within  the  Federal  Government.  We  would  like  to  ask  you  some  questions  about 
technology  forecasting  and  related  tools  and  your  perception  about  their  utility.  Participation  is 
completely  voluntary  and  you  may  refuse  to  answer  any  questions  or  withdraw  your  consent  at 
any  time.  We  would  like  to  audio-record  our  conversation  with  your  consent,  and  if  you’d  like  to 
tell  us  something  that  is  off  the  record,  we  will  stop  recording  and  writing  until  you  tell  us  that  we 
can  start  again.  Recordings  will  be  erased  after  transcribed,  and  transcripts  will  be  disposed  at  the 
end  of  the  project.  Recording  and  transcripts  will  not  be  shared  with  IARPA — only  summaries  and 
aggregate  data  will  be  shared.  May  we  have  your  pennission  to  record  the  interview  and  proceed? 

Interview  goals: 

•  To  understand  the  role  of  technology  forecasting  in  decision-making 

•  To  understand  the  use  of  existing  technology  forecasting  tools  and  their  perceived 
benefits  and  shortcomings 

•  To  understand  potential  current  and  future  needs  in  technology  forecasting  and 
technology  forecasting  tools 

IARPA  POC: 

Individual  Background: 

•  Date: 

•  Agency: 

•  Office/Department/Division: 

•  Office/Department/Division  Function: 

•  Do  you  currently  use  technology  forecasting/projections? 
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-  If  no,  how  familiar  are  you  with  technology  forecasting  and  technology  forecasting 
tool? 

Use  of  Technology  Forecasting/Projecting 

•  Does  technology  forecasting  have  a  fonnal  role  in  your  office? 

-  If  yes,  what  is  that  role? 

-  If  no,  do  you  use  technology  forecasting  and  how? 

•  Is  technology  forecasting  used  officially  in  decision-making  in  your  office? 

-  If  so,  how? 

o  What  kinds  of  forecasts  (predicting  what?  E.g.,  performance  vs.  technology)? 

-  If  not,  how  is  it  used? 

•  How  exactly  do  you  use  technology  forecasting? 

•  Are  any  automated  forecasts  used  in  decision  making  within  your  office? 

Forecasting  Tools 

•  What  kinds  of  tools  do  you  use  for  your  technology  forecasts? 

-  Specific  for  technology  forecasting? 

-  What  kinds  of  technology  are  you  interested  in  forecasting/predicting? 

-  Source? 

-  Have  you  used  any  other  tools  or  approaches  in  the  past? 

•  Can  you  tell  us  about  some  of  the  benefits  and  drawbacks  to  the  forecasts  and  the 
forecasting  tool? 

-  Broad  trends  or  specific  technologies? 

-  Time  frame  for  forecasts 

-  Precision 

-  Accuracy 

-  Interface 

-  Data  input 

-  Data  output 

-  Data  visualization 
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•  Are  you  aware  of  other  tools? 

-  If  so,  which? 

-  What  are  their  benefits/ drawbacks? 

Data  for  Technology-related  Decision  Making 

•  Do  you  regularly  use  any  data  (reports,  databases,  etc.),  outside  of  formal  technology 
forecasts,  to  aid  in  your  decision-making? 

-  If  so,  what  are  those  sources? 
o  Publicly  available? 

o  Purchased?  Source? 

-  How  are  these  sources  used  in  your  decision-making? 

•  Is  your  analysis  and/or  decision-making  limited  by  data  available?  If  so,  how? 

-  Is  there  any  other  data  out  there  that  you  can’t  access? 

-  If  not,  what  kind  of  data  do  you  need? 

•  Are  there  places  where  an  automated  tool  for  data  analysis  or  data  visualization  might 
help? 

Attitudes  about  Forecasting  Tools 

•  If  you  are  currently  using  a  tool,  how  much  do  you  trust  this  tool? 

•  If  you  stopped,  why  did  you  stop? 

-  Did  you  encounter  specific  problems? 
o  Cost? 

o  Usefulness? 
o  Interface? 
o  Data? 

•  If  you  don’t  currently  use  tools,  could  you  ever  be  convinced  to  use  a  forecasting  tool? 

-  Why  or  why  not? 

o  What  criteria  (standards)  would  the  tool  have  to  meet? 

•  Are  you  aware  of  the  kinds  of  tools  that  are  currently  available  in  technology 
forecasting? 

-  If  so,  what  is  the  source  of  your  infonnation? 


A-4 


•  If  you  don’t  currently  use  forecasting  or  forecasting  tools,  are  there  problems/questions 
that  you  currently  deal  with  that  might  benefit  from  the  use  of  technology  forecasting 
tools? 

-  What  kinds  of  questions? 

-  What  would  your  ideal  tools  be  able  to  do? 

•  If  you  don’t  plan  to  use  automated  forecasting,  why  not? 

-  What  barriers  (trust,  data  issues,  etc.)  would  you  have  to  overcome? 
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Appendix  B. 

Summaries  of  Interviews 


The  following  summaries  are  based  on  STPI  research  staffs  interviews  with  agency 
personnel  as  well  as  background  research. 

Air  Force  Office  of  Scientific  Research  (AFOSR) 

AFOSR  oversees  a  basic  research  portfolio  of  over  1200  grants  invested  globally  at 
universities,  industry  and  internal  research  programs.10  Recently,  AFOSR  began  an  effort 
in  collaboration  with  the  Basic  Research  Innovation  and  Collaboration  Center  (BRICC).  A 
partnership  with  the  Virginia  Tech  Applied  Research  Corporation  (VT-ARC),11  it  focuses 
on  the  use  of  bibliometrics  and  development  of  related  tools  to  inform  decisions  related  to 
technical  research  investment,  and  promote  a  cohesive  investment  strategy.  Analytic 
methods  are  particularly  useful  in  situations  where  resources  are  scarce  (such  as  the 
international  office),  the  investments  are  big,  the  field  is  diverse  and  the  Air  Force  can 
make  a  significant  impact  through  its  investment,  such  as  reconfigurable  electronics. 

At  this  time,  AFOSR  is  interested  in  using  analytics  to  support  their  requirement  to 
report  on  the  impact  of  their  portfolio.  AFOSR  delivers  briefs  on  their  portfolios  first  to  the 
AF  acquisition  office,  which  reports  to  the  Chief  Office  of  the  AF  (which  reports  to  the 
Congress),  and  ASD(R&E). 

AFOSR  would  like  to  increase  their  use  of  analytics  in  investment  decision-making. 12 
One  area  of  particular  interest  is  the  application  of  analytics  on  issues  of  personnel  and 
research  networks.  Since  AFOSR  invests  primarily  in  people,  they  are  interested  in  new 
ways  to  predict  on  which  young  faculty  and  rising  researchers  AFOSR  should  focus  their 
investments  to  ensure  success.  The  ability  to  identify  promising  researchers  is  seen  as 
important  as  people  represent  longer-tenn  investments  with  better  outcomes  than  research 
projects.  The  current  approach  to  identifying  new  researchers  is  to  ask  current  principle 
investigators  for  their  recommendations.  Tools  that  can  provide  a  people  forecast  by  tracing 


10  Wright  Patterson  Air  Force  Base  website,  “AFOSR — About,” 
http://www.wpafb.  af.mil/library/factsheets/factsheet.  asp?id=l  9470. 

1 1  BRiCC  website,  “BRICC  Projects,”  http://bricc.vt-arc.org/about. 

12 

~  AFOSR  had  previously  commissioned  STPI  to  examine  the  utility  of  different  tools  to  identify 
emerging  areas  of  research.  As  part  of  this  study,  AFOSR  had  STPI  research  staff  evaluate  FUSE  for 
use  by  AFOSR  program  managers. 
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how  people  coalesce  and  how  fields  grow  as  a  consequence  and  attempting  to  predict  which 
young  investigators  have  the  most  potential  for  useful  research  in  a  given  field. 

Program  mangers  currently  use  widely  available  tools  like  Web  of  Science 
(publication  database)  and  Google  Scholar.  The  Basic  Research  Innovation  Collaboration 
Center,  established  in  partnership  with  the  Virginia  Tech  Applied  Research  Corporation 
develops  tools  and  methods  for  data  analytics  that  are  used  by  AFOSR  for  portfolio 
analysis  and  analyzing  trends  in  science  and  technology.  However,  the  program  managers 
still  rely  heavily  on  subject  matter  experts  to  understand  current  and  future  states  of 
technology.  Tools  to  assist  in  reporting,  information  capture  and  the  ecosystems  of 
communication  were  suggested  as  useful  directions  to  consider. 

The  respondents  that  STPI  research  staff  spoke  with  believe  that  a  forecast  should  be 
a  hypothesis  to  be  tested,  rather  than  the  final  word.  They  also  believe  that  there  should 
always  be  a  human  in  the  loop  to  interpret  forecast  results  for  policymakers  as  the  “burden 
of  proof  is  so  much  higher  on  machines”  At  present,  one  of  the  barriers  to  using  tool  to 
provide  input  into  decision-making  is  lack  of  trust  in  automated  forecasts.  Trust  needs  to 
be  built  and  earned  at  the  management  level.  One  way  to  build  this  trust  is  to  make  the 
levels  of  uncertainty  transparent  by  embedding  uncertainty  quantification  in  the 
methodology  or  other  methods  to  ensure  that  the  wrong  message  does  not  get  presented  to 
higher  level  decision-makers. 

The  respondents  also  mentioned  that  there  is  fear  that  a  forecasting  tool  will  replace 
a  program  officer.  However,  those  who  champion  automated  methods  believe  that  they 
could  be  used  to  support  the  analyst  and  as  a  check  against  cognitive  bias  and  lend 
credibility  to  decisions. 

Central  Intelligence  Agency  (CIA),  Directorate  of  Science  and 
Technology 

The  official  at  the  CIA’s  Directorate  of  Science  and  Technology  (DS&T)  interviewed 
for  this  task  believes  that  the  CIA  uses  technology  forecasting,  although  not  adequately. 
The  CIA  is  a  customer  for  science  and  technology  intelligence  forecasting  and  geopolitical 
forecasting  where  S&T  plays  a  role.  Forecasts  are  used  to  plan  research  investments, 
tactically  and  strategically.  The  planning  follows  a  3 -year  budget  cycle  starting  with  a 
major  issue  study.  In  year  3,  there  is  a  consolidated  intelligence  investment  strategy 
followed  by  a  congressional  budget  justification. 

There  is  no  in-house  tool  development  within  DS&T;  they  believe  that  the 
government  is  best  served  by  finding  a  niche  where  the  private  sector  cannot  or  will  not 
play,  rather  than  being  in  competition  with  the  private  sector.  In-Q-Tel,  the  CIA’s  venture 
fund,  is  a  conduit  to  private  sector  startups.  They  invest  in  dual-use  tools  that  are  being 
developed  for  commercial  sector  marketing  and  commercial  sector  forecasts.  In-Q-Tel  is 
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looking  at  capabilities  of  companies  such  as  Recorded  Future,  which  use  machine-learning 
based  forecasting  approaches.13  Lab41,  an  In-Q-Tel  owned  and  operated  venture,  is  a 
testing  ground  for  applying  emerging  analytic  techniques  to  intelligence  community 
problems. 14  These  tools  are  sourced  by  the  CIA’s  Open  Source  Center  for  forecasting,  and 
used  by  agency  analysts.  The  analysts  are  seen  as  essential  to  the  process,  as  they  come  in 
to  close  the  loop  on  answering  the  “so  what?”  question. 

Defense  Threat  Reduction  Agency  (DTRA),  Research  and  Development 
Directorate  (J9) 

DTRA’s  mission  is  to  safeguard  the  United  States  and  its  allies  from  weapons  of  mass 
destruction  (chemical,  biological,  radiological  and  nuclear)  and  high  yield  explosives  by 
developing  capabilities  to  reduce,  eliminate  and  counter  the  threat. 15  J9  is  DTRA’s  research 
and  development  division,  and  funds  activities  ranging  from  basic  research  through 
development  and  integration  of  technologies.16  The  types  of  decisions  they  are  trying  to 
inform  are  whether  research  should  be  initiated  in  an  area,  an  appropriate  level  of 
monitoring  for  a  potential  threat,  preventing  future  WMD  threats  etc.  They  look  at  things 
on  two  scales,  intent  and  capability. 

One  aspect  of  DTRA  J9’s  role  is  to  anticipate  emerging  challenges  to  be  able  to 
provide  timely  responses  to  their  customers,  who  are  combatant  commands  and  services. 
The  combatant  commands  and  services  informs  DTRA  J9  about  their  current  needs,  but 
this  introduces  a  lag,  since  ideally,  DTRA  would  be  able  to  address  potential  threats  before 
they  become  a  problem. 

Forecasting  and  prediction  are  seen  as  a  challenging  activities  associated  with  many 
limitations.  Much  of  the  infonnation  available  today  is  based  on  anecdotal  evidence  and 
not  analytically  driven.  Instead  of  predictions,  DTRA  focuses  on  approaches  for  looking  at 
observable  trends.  The  thought  is  that  once  you  document  what  you  are  seeing  from  a  trends 


13 

Recorded  Future  website,  “Out-lnnovate  Your  Adversaries  with  Real-Time  Threat  Intelligence,” 
https://www.recordedfuture.com/. 

14  Lab41  is  a  testing  facility  for  big  data  analytics  that  is  operated  by  the  non-profit  funding  arm  of  the 
CIA  called  In-Q-Tel.  The  operation  located  in  Silicon  Valley  is  unclassified.  It  was  designed  to  enable 
private  sector  software  developers  and  engineers  to  collaborate  on  developing  techniques  to  apply 
analytics  to  massive  amounts  of  data  for  the  IC.  According  to  Lab41,  projects  address  data 
visualization,  social  network  analysis  and  statistical  modeling  of  structured  and  unstructured 
information.  “Challenge  teams”  operate  out  of  a  facility  based  in  Menlo  Park,  California,  working  on 
selected  projects  within  a  3-12-month  time  frame.  Lab41  website,  “Where  Experts  in  Needles  & 
Haystacks  Work  Together,”  https://www.lab41.org/. 

15  Defense  Threat  Reduction  Agency  website,  “Missions,”  http://www.dtra.mil/Missions.aspx. 

16  Global  Biodefense  website,  “DTRA  Seeks  Counter-WMD  Research  Support  Services,” 
http://globalbiodefense.com/2014/08/28/dtra-seeks-counter-wmd-research-support/. 
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perspective,  you  can  do  an  analysis.  Essentially,  they  would  like  to  take  a  Net  Assessment17 
and  convert  that  into  “news  you  can  use.” 

DTRA  has  been  brainstonning,  for  several  years,  on  ways  to  integrate  different 
approaches;  no  single  methodology  or  approach  has  everything  they  need.  The  process  they 
would  like  to  have  in  place  is  to  (1)  look  at  trends  that  are  either  clearly  impactful  or 
possible  impactful,  (2)  assess  those  trends,  (3)  have  DTRA  and  the  CWMD  community  of 
practice  look  at  the  assessments,  (4)  inform  intelligence  collectors,  (5)  circle  back  and 
iterate  with  people  in  the  IC. 

DTRA  J9’s  goal  is  to  institutionalize  technology  forecasting  and  they  are  open  to 
leveraging  IARPA’s  activities.  Within  their  context,  an  automated  forecasting  tool  would 
be  used  in  the  following  ways  (1)  to  validate  known  infonnation,  (2)  to  identify  new 
application  of  existing  technology,  and  (3)  to  tie  less  known  technologies  with  new 
applications.  Their  ideal  forecast  would  look  at  a  timescale  of  5-15  years  out.  The 
expectation  is  that  the  tool  provide  defensible  results  that  are  analytically  sound; 
transparency  is  a  requirement.  Any  certainty  greater  than  50%  would  be  acceptable  to  them. 

Department  of  Defense  (DOD),  Office  of  Technical  Intelligence, 

Assistant  Secretary  of  Defense  for  Research  and  Engineering 
(ASD|R&E|) 

The  Office  of  Technical  Intelligence  analyzes  global  science  and  technology  activities 
to  inform  future  DOD  investments  in  workforce,  infrastructure  and  research  funding.  The 
OTI  conducts  technology  watch  and  horizon  scanning  activities  with  the  following  goals: 
(1)  “identify[ing]  emerging  S&T  areas  that  will  enhance  operational  capability  in  the  next 
10-20  years,”  (2)  “assess [ing]  scientific  areas  that  should  be  included  in  the  long-range 
(10+  years)  research  strategy,”  and  (3)  “provid[ing]  context  to  understand  the  S&T  that 
will  result  in  significant  new  discoveries. 18 

The  respondents  at  OTI  believe  that  decisions  today  are  not  being  informed  by  data 
analytics  or  technology  forecasting  to  any  great  extent.  They  believe  that  people  are  still 
seeking  help  to  understand  the  current  state  of  things,  which  analytics  can  help  answer. 
Accordingly,  they  are  focusing  not  on  data-driven  decisions  but  data-enabled  decisions. 
Their  current  focus  is  on  developing  analytics  for  basic  science.  These  analytics  would  be 
used  by  program  managers  to  understand  gaps  and  emerging  areas  in  their  investment 
portfolio,  centers  of  excellence,  as  well  as  to  identify  lead  researchers  and  young  and  rising 
researchers  in  their  area.  (Automated  tools  are  used  as  part  of  the  process,  not  all  of  it;  tool 


17 

Net  Assessments  are  intended  to  look  long  term  (20-30  years)  into  the  military’s  future. 

18 

Department  of  Defense  Research  &  Engineering  Enterprise  website,  “Science  &  Technology  Corner, 
Technology  Watch  and  Horizon  Scanning  for  the  Department  of  Defense,” 
http://www.acq.osd.mil/chieftechnologist/cto/cto_TWHS.html. 
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results  are  supplemented  by  inputs  from  subject  matter  experts  which  are  then  used  by 
analysts  to  interpret  such  that  the  resulting  infonnation  is  relevant  to  the  perspective  of  a 
decision-maker).  Their  methodologies  are:  (1)  horizon  scanning,  the  detection  of  changes 
and  emergence,  despite  a  weak  signal  to  noise  ratio,  and  (2)  technology  watching,  the 
collection  all  possible  information  on  a  technology,  actionable  intelligence.  OTI  leverages 
commercial  technology  forecasting  systems  as  well  as  invests  in  the  development  of  new 
forecasting  approaches  (such  as  TechWatch)  that  integrate  multiple  data  analysis  streams 
and  a  suite  of  S&T  forecasting  tools. 

In  the  future,  OTI  will  focus  on  identifying  enabling  technologies  for  a  future 
capability.  Their  approach  for  this  is  to  identify  a  capability  of  interest,  break  it  down  to 
the  technologies,  break  it  out  to  the  components  and  start  with  developing  a  taxonomy  for 
that  level. 

The  respondents  see  forecasting  as  an  iterative  process  with  refinements  from  SMEs; 
the  idea  is  for  the  data  to  queue  up  the  SME.  They  see  their  decision-making  structure  as 
consisting  of  the  output  of  an  automated  process  combined  with  subject  matter  expertise 
providing  infonnation  to  the  interpreter  who  provides  their  input  to  a  decision-maker  (such 
as  the  Director  of  the  Office  of  Technical  Intelligence)  who  then  provides  it  to  the  higher 
level  decision-maker  (such  as  the  Assistant  Secretary  of  Defense  for  Research  and 
Engineering). 

The  respondents  see  the  role  of  humans  “in  the  loop”  as  necessary  to  provide  context 
and  validation  to  the  question  and  tool  output  also  the  presence  of  a  human  helps  to  build 
trust  in  the  forecast,  as  in  their  experience,  people  trust  people,  not  tools. 

This  group  does  not  see  long-range  forecasts  (beyond  5  years)  as  within  their 
capabilities;  they  are  targeting  the  2-5-year  time  frame. 

Energy  Information  Administration  (EIA) 

EIA  produces  technology  forecasts  for  energy  demand  by  sector  and  type  of  fuel.  The 
forecasts  are  based  on  the  National  Energy  Modeling  System,  which  is  an  integrated  model 
of  the  U.S.  Energy  demands  linked  to  a  macroeconomic  model,  developed  internally  at 
EIA.  Inputs  to  the  model  include  resource  availability,  technological  choice  and 
characteristics,  demographics  and  global  energy  market  interactions.  For  newer 
technologies  with  less  established  markets,  historical  precedent  and  limiting  factors  are 
factored  into  the  projections. 

The  EIA’s  primary  product  is  the  Annual  Energy  Outlook,  which  is  published  for 
external  customers  including  the  Congress.  Forecast  timelines  are  typically  1  to  2  years, 
but  projections  for  longer  timescales  are  also  conducted.  Projections  are  provided  at  the 
Census  level  of  detail.  Sensitivity  analysis  using  test  cases  is  part  of  the  modeling  activity 
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Government  Accountability  Office  (GAO) 

GAO  prepares  technology  assessments  of  current  and  emerging  technologies  to 
understand  implications,  challenges  and  opportunities  for  “Federal  agencies  and 
departments  and  their  wider  impact  on  American  society.”19  Topics  for  these  studies  are 
based  on  current  and  emerging  technologies  identified  by  Congress.  The  studies  look  at  the 
anticipated  trajectory  of  the  technology,  cost  factors  and  policy  implications  including 
export  control,  IP  and  liability.  GAO  reports  and  recommendations  are  for  the  government, 
but  Congress,  not  the  GAO,  makes  decisions  based  on  their  technology  assessments. 

The  GAO  technology  assessment  staff  do  not  use  any  automated  tools.  They  rely  on 
literature  searches  and  reviews,  searches  by  library  staff,  and  interviews  with  experts.  An 
important  focus  of  these  studies  is  the  global  economic  impact  of  a  technology,  and  this  is 
challenging  to  forecast.  Therefore,  they  do  more  cost  and  economic  modeling  more  than 
technology  modeling. 

The  respondents  said  that  they  would  find  it  useful  to  identify  historical  patterns  such 
as  those  for  general  purpose  technologies  like  electricity,  and  apply  those  patterns  to 
current  technologies  and  technology  enablers.  Some  tools  they  have  liked  are  the  European 
Parliamentary  Team  Assessment  (EPTA),  a  mind  mapping  tool,  which  helps  a  team  build 
connections  between  topics.  Issues  they  see  with  tools  are  the  signal-to-noise  ratio,  and  the 
number  of  technologies  that  have  emerged  very  rapidly.  They  feel  the  text  analytics  and 
clustering  tools  need  validation,  but  ultimately  would  like  to  have  a  data  analytics 
approach. 

National  Institutes  of  Health,  National  Institute  of  Biomedical  Imaging 
and  Bioengineering  (NIBIB) 

The  mission  of  NIBIB  is  to  improve  health  through  research  and  development  of  new 
biomedical  imaging  and  bioengineering  techniques  and  devices  to  improve  disease 
detection,  prevention  and  treatment.20  Approximately  75-80%  of  the  research  funded  by 
NIBIB  is  investigator  driven.  The  remaining  research  topics  come  from  a  variety  of  sources 
ranging  from  top-down  direction  from  the  White  House  to  bottom-up  needs  based 
initiatives  such  as  platfonn  technologies  for  mobile  health  research.  NIBIB  also 
collaborates  with  NSF  and  DOE. 

The  majority  of  the  technology  identification  and  assessment  work  done  through 
portfolio  analysis,  comparing  the  current  state  of  what  is  being  funded  and  with  program 


19  U.S.  Government  Accountability  Office  website,  “Technology  Assessment,” 
http://www.gao.gov/technology_assessment/key_reports. 

20 

“  NIH  Almanac  website,  “National  Institute  of  Biomedical  Imaging  and  Bioengineering  (NIBIB) — 
Mission,”  http://www.nih.gov/about-nih/what-we-do/nih-almanac/national-institute-biomedical- 
imaging-bioengineering-nibib. 
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officers  primarily  responsible  for  identifying  emerging  areas  in  need  of  funding  or 
additional  funding.  InSpire  and  bibliometric  tools  are  generally  used  for  these  analyses. 
Once  potential  areas  are  identified,  subject  matter  experts  (usually  current  or  previously 
funded  investigators)  experts  or  formal,  public  requests  for  information  (RFIs)  to  identify 
research  areas  that  are  challenges  that  the  institute  should  tackle.  Occasionally,  crowd¬ 
sourcing  is  used  to  gather  infonnation  and  ideas. 

Several  different  types  of  tools  were  suggested  as  being  potentially  useful  to  the 
NIBIB  decision-making  process:  (1)  tools  for  analyzing  what  factors  lead  to  successful 
research,  (2)  a  predictive  tool  that  is  able  to  replicate  peer  review  results  and  then  do  better 
than  peer  review,  (3)  tools  for  identifying  critical  people  in  research  networks  (there’s  a 
limited  grasp  of  the  biomedical  workforce),  (4)  tools  to  provide  information  on  how  grants 
come  together,  and  (5)  tools  that  can  assist  in  allocating  money  in  more  efficiently  than  is 
done  currently  (between  short-term  and  long-term  research). 

The  respondent  believes  that  a  human  “in  the  loop”  is  essential  to  guide  the  results  of 
data  mining  and  provide  context.  One  of  the  barriers  to  adoption  is  that  the  underlying 
approaches  in  automated  forecasting  tools  are  statistically  based,  and  not  informed  by 
context.  There  is  doubt  that  such  an  approach  would  be  better  than  their  current 
methodology  for  identifying  technologies  of  interest. 

Office  of  the  Director  of  National  Intelligence  (ODNI),  Acquisition, 
Technology  and  Facilities 

The  respondent  within  Acquisition,  Technology,  and  Facilities  at  ODNI  believes  it  is 
not  clear  that  technology  forecasting  plays  a  role  in  meeting  the  needs  of  fonnal  policy 
makers  (such  as  the  National  Security  Council)  or  helpful  in  developing  fonnal  policies 
such  as  crafting  Executive  Orders  and  Presidential  Policy  Directives).  Decisions  made  at 
this  level  are  characterized  by  significant  internal  debate  among  many  people,  and  a  human 
analyst  is  needed  to  sort  through  issues  and  provide  interpretation  and  context  around 
information  needs — answering  questions  such  as  “What  does  this  mean?  Why  should  I 
care?  What  should  I  do?” 

The  respondent  does  not  currently  see  any  entry  points  for  automated  technology 
forecasting  in  the  decision-making  chain,  and  is  not  sure  when  this  point  will  be  reached. 
However,  they  do  see  a  role  in  machine  sorting  of  infonnation  to  support  an  analyst,  but 
not  creating  insight. 

The  office  has  explored  the  use  of  network  analysis  tools  such  as  Palantir  and  Gephi 
and  they  are  comfortable  with  the  methodology  used  by  the  tools.  The  interpretation  still 
needs  to  be  done  by  the  analyst. 

The  respondent  at  ODNI  believes  that  trust  and  credibility  of  tools  (relative  to  human 
analysts)  are  important  issues.  Decision-makers  have  to  be  comfortable  with  the 


B-7 


uncertainty  level  of  automated  forecasts.  Tool  developers  have  to  also  think  about 
transparency  and  accountability.  There  are  big  challenges  to  be  met  for  inserting  automated 
forecasting  in  the  decision-making  process,  and  focusing  heavily  on  delivering  a  tool  that 
is  easily  transferrable  will  not  help  accomplish  this. 
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Appendix  C. 

Technology  Forecasting  Tools, 
Techniques,  and  Aids 


Forecasting  Methods 

•  Backcasting:  In  backcasting,  one  begins  with  envisioning  a  particular  future 
state,  and  then  traces  this  state  backwards  in  time  to  the  present  to  determine  the 
pathways,  events,  or  conditions  that  could  lead  to  this  future  state  (NRC  2010). 

•  Horizon  Scanning:  horizon  scanning  systems  systematically  gather  infonnation 
to  inform  policy  makers  about  emerging  issues  (Habegger  2009).  For 
technology  forecasting  applications,  horizon  scanning  refers  to  tools  or 
techniques  that  provide  advance  warning  or  early  indications  of  emerging 
technologies. 

•  Prediction  Markets:  In  prediction  markets,  the  collective  judgement  of  group  of 
individuals  are  used  to  generate  a  forecasts  about  an  event  or  parameter. 
Predictions  are  treated  “as  assets  to  be  traded  on  a  virtual  market  that  can  be 
accessed  by  a  number  of  individuals. . . .  The  final  market  value  of  the  asset  is 
taken  to  be  indicative  of  its  likelihood  of  occurring”  (Habegger  2009) 

•  Scenarios:  scenarios  are  “stories  about  alternative  futures  focused  on  the 
forecasting  problem  at  hand”  (NRC  2010),  which  are  used  to  understand  how 
different  factors  influence  a  future  state.  In  technology  forecasting  applications, 
scenarios  are  often  used  to  examine  the  different  possible  development  pathways 
for  a  technology.  They  can  also  be  used  to  examine  the  different  technological 
capabilities  desired  in  different  future  states. 

•  Trend  Analysis:  In  trend  analysis,  a  user  or  tool  uses  infonnation  the  past  and 
present  to  forecast  a  future  state.  These  techniques  assume  that  “the  future 
represents  a  logical  extension  of  the  past  and  that  predictions  can  be  made  by 
identifying  and  extrapolating  the  appropriate  trends  from  the  available  data” 
(NRC  2010). 

Tools,  Capabilities,  and  Programs 

•  Forecasting  Science  and  Technology  (ForeST):  ForeST  is  an  IARPA  program 
whose  goal  is  to  “develop  and  test  methods  for  generating  accurate  forecasts  for 
significant  science  and  technology  (S&T)  milestones,  by  combining  the 


C-l 


judgments  of  many  experts.”21  As  part  of  the  ForeST  program,  researchers  were 
funded  to  run  SciCast,  described  below. 

•  Foresight  and  Understanding  from  Scientific  Exposition  (FUSE):  FUSE  is  an 
IARPA  program  whose  goal  is  to  “mitigate  technology  surprise  by  the  reliable 
and  early  detection  of  emerging  scientific  and  technological  concepts  across 
multiple  disciplines  through  automated  analysis  of  the  primary  scientific 
literature”  (Murdick  n.d.).  FUSE  forecasts  emerging  scientific  and  technological 
concepts  by  predicting  terms  (unigrams  to  4-grams)22  that  are  likely  to  increase 
in  usage  within  the  next  2-3  years. 

•  IN-SPIRE™:  IN-SPIRE  is  a  software  tool  developed  by  Pacific  Northwest 
National  Laboratory  to  organize  and  visualize  information  from  unstructured 
text  documents.  It  “determines  key  topics  or  themes  in  each  to  create  a  signature 
for  each  document  in  the  collection.  IN-SPIRE’s  two  main  visualizations  display 
representations  of  the  documents  in  which  those  with  similar  or  related  topics 
appear  closer  together.”23 

•  SciCast:  SciCast  was  the  largest  science  and  technology  tournament  program.24 
It  was  run  by  George  Mason  University  from  November  2014  through  May 
2015  as  part  of  IARPA’s  Forecasting  Science  and  Technology  (ForeST) 
program.25  Participants  could  pose  and  answer  questions  related  to  science  and 
technology  developments.  Participants  could  change  their  forecasts  at  any  time 
based  on  new  infonnation,  which  made  the  SciCast  platform  a  real-time 
indicator  of  what  participants  think  is  going  to  happen  (Tsarchopoulos  2014). 

•  Science  of  Science  (Sci2):  Sci2  is  a  tool  developed  by  Katy  Borner  and  Kevin 
W.  Boyack  of  SciTech  Strategies  Inc.  to  study  science  research  and  practice.  It 
allows  users  to  input  text  and  run  multiple  types  of  analyses,  including  temporal 
analysis,  geospatial  analysis  and  mapping,  topical  analysis  and  mapping,  tree 
analysis  and  visualization,  and  network  analysis  and  visualization  (Sci2  Team. 
2009). 


21  IARPA  website,  “Forecasting  Science  &  Technology  (ForeST),” 
http://www.iarpa.gov/index.php/research-programs/forest. 

22 

An  n-gram  is  a  sequence  of  adjacent  words  from  a  document.  A  unigram  is  made  up  of  a  single  word 
and  a  bigram  is  a  single  word  plus  the  word  that  follows  it.  An  n-gram  is  a  sequence  of  a  word  plus  the 
n-1  words  that  follow  it.  In  general,  larger  values  of  n  give  greater  context  to  the  string  of  terms. 

23  IN-SPIRE  website,  “About  IN-SPIRE,”  http://in-spire.pnnl.gov/about.stm. 

24  IARPA  website,  “Forecasting  Science  &  Technology,”  http://www.iarpa.gov/index.php/research- 
programs/ forest. 

25  The  Official  SciCast  Blog,  “About,”  http://blog.scicast.org/about/. 


C-2 


•  Technology  Watch/Horizon  Scanning  (TW/HS):  This  is  a  web  tool  currently 
under  development  by  OTI  within  the  Office  of  ASD(R&E)  to  (1)  identify 
emerging  S&T  areas  that  will  enhance  operational  capability  in  the  next  10-20 
years,  (2)  assess  scientific  areas  that  should  be  included  in  the  long-range  (10+ 
years)  research  strategy,  and  (3)  provide  context  to  understand  the  S&T  that  will 
result  in  significant  new  discoveries.26  It  will  leverage  algorithmic  approaches, 
commercial  systems,  and  infonnation  on  research  investments  and  outputs. 


26 

"  Department  of  Defense  Research  &  Engineering  Enterprise  website,  “Science  &  Technology  Corner, 
Technology  Watch  and  Horizon  Scanning  for  the  Department  of  Defense,” 
http://www.acq.osd.mil/chieftechnologist/cto/cto_TWHS.html. 
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Appendix  D. 

Case  Studies  of  Federal  Agencies 


The  IDA  researchers  conducted  case  studies  of  three  Federal  agencies  selected  from  the  set 
of  agencies  examined  because  they  each  present  a  distinct  opportunity  for  technology  forecasting 
tools  to  help  improve  the  agencies’  current  processes.  The  three  selected  for  case  study  were: 

•  Agency  for  Healthcare  Research  and  Quality  in  the  Department  of  Health  and  Human 
Services 

•  Bureau  of  Industry  and  Security  in  the  Department  of  Commerce,  and 

•  Chief  of  Naval  Operations  Strategic  Studies  Group 

The  case  studies  consisted  of  background  research  by  IDA  researchers  and  infonnation  from 
structured  conversations  between  IDA  researchers  and  representatives  of  the  agencies. 

Agency  for  Healthcare  Research  and  Quality  (AHRQ) 

Background  and  Goals 

AHRQ  within  the  U.S.  Department  of  Health  and  Human  Services  runs  the  Effective  Health 
Care  Program  and  funds  “effectiveness  and  comparative  effectiveness  research  for  clinicians, 
consumers,  and  policymakers.”27  The  goal  of  the  research  is  to  inform  health-care  decisions,  such 
as  clinical  guidelines,  health  care  policies,  and  insurance  coverage  decisions  (AHRQ  2014).  The 
Effective  Health  Care  Program  has  seven  steps  to  conduct,  sustain,  and  advance  the  comparative 
effectiveness  research,28  and  this  case  study  examines  AHRQ’s  process  for  the  first  of  these  steps, 
identifying  new  and  emerging  clinical  interventions,  which  is  done  through  the  AHRQ  Healthcare 
Horizon  Scanning  System.  The  six  remaining  steps  are  not  discussed  since  they  are  not  related  to 
technology  forecasting. 

The  Healthcare  Horizon  Scanning  System  has  been  funded  since  2009  to  inform  patient- 
centered  outcomes  research  investments  through  scanning  emerging  healthcare  technologies  and 


27 

~  AHRQ,  Effective  Health  Care  Program  website,  “What  Is  the  Effective  Health  Care  Program,” 
http://effectivehealthcare.ahrq.gov/index.cfm/what-is-the-effective-health-care-programl/. 

28 

“  AHRQ,  Effective  Health  Care  Program  website,  “What  Is  Comparative  Effectiveness  Research,” 
http://effectivehealthcare.ahrq.gov/index.cfm/what-is-comparative-effectiveness-researchl/. 
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innovations.29  AHRQ’s  system  is  the  first  “publicly  available,  comprehensive  system. .  .for  [health 
care]  horizon  scanning  in  the  United  States”  (DeLurio  et  al.  20 15). 30  The  goal  of  the  system  is  to 
identify  emerging  technologies  or  innovations  that  could  meet  previously  unmet  health  needs. 
Examples  of  emerging  health  care  technologies  and  innovations  include  “new  (and  new  uses  of 
existing)  pharmaceuticals,  medical  devices,  interventions,  behavioral  health  interventions,  health 
care  delivery  innovations,  and  public  health  and  health  promotion  activities.”  Originally,  the  output 
of  the  Healthcare  Horizon  Scanning  System  was  intended  to  be  used  for  research  prioritization. 
However,  in  2010,  the  Patient-Centered  Outcomes  Research  Institute  (PCORI)  was  created  to  fund 
comparative  clinical  effectiveness  research  with  the  goal  of  detennining  “which  of  the  many 
healthcare  options  available  to  patients  and  those  who  care  for  them  work  best  in  particular 
circumstances.”31  As  a  research  funding  organization,  PCORI  determines  its  own  priorities, 
though  it  may  use  the  output  of  the  Healthcare  Horizon  Scanning  System. 

Process  Overview 

The  Healthcare  Horizon  Scanning  System  is  a  non-automated  process,  the  details  of  which 
are  shown  in  Figure  D-l.  Additional  details  follow.  Medical  librarians  search  daily  for  leads  from 
publicly  available  sources  such  as  medical  journals,  scientific  literature,  company  press  releases, 
trade  publications,  and  newsletters.  A  set  of  criteria  is  used  to  guide  the  selection  of  leads,  which 
must  be  related  to  AHRQ’s  priority  areas  or  a  cross-cutting  area.  Based  on  the  relevant  AHRQ 
priority  area,  subcategory,  and  topic  class,  as  well  as  whether  it  meets  a  set  of  criteria,  leads  are 
sorted  into  a  list  of  possible  topics.  Analysts  review  the  potential  leads  and  nominate  topics,  which 
are  discussed  and  voted  upon  by  a  larger  group  that  includes  analysts,  librarians,  content  team 
leader,  project  manager,  and  other  invited  staff  and  experts.  Topics  that  receive  a  majority  of  votes 
are  entered  into  the  Healthcare  Horizon  Scanning  System.  Once  in  the  system,  a  topic  is  assigned 
to  an  analyst  with  relevant  expertise  for  more  in-depth  research  and  to  a  medical  librarian  for  a 
detailed  topic  profile.  The  topic  profiles  are  reviewed  by  internal  and  external  research  experts. 
Following  this  review  process,  high-impact  topics  are  selected  and  written  up  in  a  report.  While 
the  high-impact  topics  only  those  that  have  late-phase  data,  additional  topics  are  in  earlier  stages 
of  research.  These  additional  topics  and  the  high-impact  topics  are  tracked  continually,  with  the 
use  of  automated  alerts  where  possible.  Topics  are  tracked  from  1  year  prior  to  Food  and  Drug 
Administration  (FDA)  approval  to  2  years  following  FDA  approval. 


29 

“  AHRQ,  Effective  Health  Care  Program  website,  “AHRQ  Healthcare  Horizon  Scanning  System,” 

http://effectivehealthcare.ahrq.gov/index.cfm/who-is-involved-in-the-effective-health-care-programl/ahrq- 

horizon-scanning-system/. 
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EuroScan  International  Network  is  a  European  healthcare  horizon  scanning  system. 

3 1 

PCORi  website,  “About  Us,”  http://www.pcori.org/about-us. 
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•BROAD  SCANNING  (Daily):  Searchers  conduct  broad  scanning  and  select  leads  for  Horizon 
Scanning  Analysts  to  consider  for  potential  topics 


•LEAD  REVIEW  AND  TOPIC  IDENTIFICATION  (Weekly):  Analysts  review  leads  in  assigned 
priority  areas  to  collect  basic  information  on  topics  that  appear  to  meet  inclusion  criteria  for 
the  Horizon  Scanning  System 

•TOPIC  NOMINATION  MEETINGS  (Monthly):  At  Topic  Nomination  Meetings  analysts  present 
potential  topics  to  Horizon  Scanning  Team  to  discuss  and  vote  on  whether  topics  meet 
inclusion  criteria  to  become  Target  Topics  for  tracking  and/or  in-depth  profile  development 

•STATUS  UPDATE  REPORT  (Five  times  annually):  Horizon  Scanning  team  reviews  all 
active  Target  Topics  to  update  development  status;  prepares  Status  Update  report;  archives 
topics  that  no  longer  meet  inclusion  criteria 

•TOPIC  PROFILE  DEVELOPMENT  (Ongoing):  Target  Topics  that  appear  to  have  late-phase 
efficacy  data  undergo  detailed  searches  by  librarians  to  enable  analysts  to  prepare  in-depth 
Target  Topic  profiles;  topics  with  no  late-phase  data  are  tracked  until  such  data  accumulate 

•EXPERT  COMMENT  PROCESS  (Ongoing):  Horizon  Scanning  analysts  suggest  experts 
from  whom  to  seek  comments;  Medical  Editor  sends  in-depth  Target  Topic  profiles  to  obtain 
expert  comments  on  high  impact  potential 


•HIGH  IMPACT  TOPIC  SELECTION  PROCESS  (Twice  annually):  Horizon  Scanning 
Analysts  read  expert  comment  sets  received  for  the  Target  Topic  profiles  sent  out  for 
comment  in  their  assigned  priority  areas  to  prepare  for  Potential  High  Impact  report  meeting; 
Team  convenes  to  discuss  expert  comments  received  in  prior  12  months  and  determine 
which  topics  experts  deemed  to  have  potential  for  High  Impact  and  identifies  topics  to 
archive  that  no  longer  meet  criteria 

•HIGH  IMPACT  REPORT  PREPARATION  (Twice  annually):  Team  writes  up  analysis  of 
Target  Topics  that  received  expert  comments  for  15-chapter  Potential  High  Impact 
Interventions  report 


•TOPIC  MONITORING  AND  UPDATING  PROCESS  (Ongoing):  Analysts  review  active  topic 
list  to  prioritize  new  and  updated  existing  Target  Topics  for  in-depth  profiles  for  next  round  of 
expert  comments 


•  INDEXING  PROCESS  (Ongoing):  Medical  indexers  assign  appropriate  controlled  vocabulary 
terms  from  medical  and  heath  care  nomenclature  systems  to  in-depth  Target  Topic  profiles 


Source:  DeLurio  et  al.  (2015). 

Figure  D-1.  Overview  of  the  AHRQ  Healthcare  Horizon  Scanning  System  Process 


While  AHRQ  does  not  make  policy  decisions,  the  outcome  of  its  work  is  used  in  decision¬ 
making,  and  healthcare  insurance  providers  have  cited  AHRQ  research  in  their  coverage  plans. 


Interest  in  Technology  Forecasts 

The  Healthcare  Horizon  Scanning  System  provides  a  short-term  forecast  by  looking  at 
innovations  approximately  1  year  before  FDA  approval  through  2  years  after  FDA  approval. 
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However,  AHRQ  is  interested  in  being  able  to  predict  further  into  the  future  to  forecast  innovations 
in  medicine. 

The  AHRQ  respondent  expressed  interest  in  a  system  that  uses  crowdsourcing.  For  example, 
Medscape  has  recently  developed  Medscape  Consult,  an  “online  community  for  physicians  to  ask 
and  answer  clinical  questions  and  share  and  discuss  clinical  challenges.”32  While  Medscape  is  not 
predictive,  the  idea  of  having  a  wider  community  nominate  topics  and  contribute  opinions  is  of 
interest.  The  respondent  also  expressed  interest  in  two  different  types  of  automated  tools:  (1) 
automated  search  capabilities  and  (2)  automated  horizon  scanning  tools  that  would  allow  someone 
to  find  out  about  more  about  a  topic  of  interest. 

Discussion 

The  AHRQ  Healthcare  Horizon  Scanning  System  is  the  most  systematic  horizon  scanning 
system  among  those  that  STPI  research  staff  encountered  in  the  government.  With  the  exception 
of  automated  alerts  on  specific  search  terms,  this  system  is  manual;  it  relies  on  librarians,  analysts, 
and  subject  matter  experts,  which  is  extremely  labor-intensive.  The  guiding  questions  and  criteria 
that  help  the  librarians  and  analysts  illustrate  the  amount  of  human  judgment  required,  which  is 
viewed  as  necessary  to  the  process.  All  of  the  steps  from  lead  identification  to  indexing  of  terms 
require  some  knowledge  of  context. 

A  tool  that  could  be  useful  to  AHRQ  would  be  one  that  automates  a  subset  of  the  steps  in  the 
existing  process  (a  tool  that  scans  the  literature,  a  tool  that  groups  leads  into  topics,  a  tool  that 
assigns  controlled  vocabulary,  etc.).  It  would  also  be  useful  to  develop  tools  that  could  reduce  bias 
in  the  process  by  suggesting  relevant  experts  for  a  given  topic,  for  example.  Currently,  analysts 
and  experts  are  used  to  determine  whether  a  topic  is  high  impact  and  to  analyze  those  high-impact 
topics.  One  step  towards  an  automated  system  would  be  to  have  an  algorithm  identify  whether  a 
topic  is  potentially  high  impact,  and  have  analysts  and  experts  review  that  proposal. 

Another  way  to  improve  the  horizon  scanning  system  would  be  to  develop  predictive 
capabilities  that  are  able  to  scan  more  early-stage  research  and  detennine  the  areas  of  patient-care 
that  could  be  affected  rather  than  looking  at  research  that  is  near  or  at  the  point  of  late-stage  trials. 
This  could  allow  for  earlier  research  investment  in  useful  health  care  technologies  or  innovations. 

Bureau  of  Industry  and  Security  (BIS) 

Background  and  Goals 

The  mission  of  BIS  is  to  “advance  U.S.  national  security,  foreign  policy,  and  economic 
objectives  by  ensuring  an  effective  export  control  and  treaty  compliance  system  and  promoting 


32 

'  Medscape  website,  “Medscape  Consult,”  http://www.medscape.com/public/consult-feature. 
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continued  U.S.  strategic  technology  leadership.”33  BIS  is  “charged  with  the  development, 
implementation  and  interpretation  of  U.S.  export  control  policy  for  dual-use  commodities, 
software,  and  technology.”34  As  part  of  this  role,  BIS  needs  to  first  detennine  emerging 
technologies  that  are  “dual-use”  or  have  both  civil  application  and  applications  related  to  the 
military,  terrorism,  or  weapons  of  mass  destruction  (15  CFR,  Subtitle  B,  Chapter  VII,  Subchapter 
C,  Part  730,  Section  730.3),  and  then  detennine  appropriate  export  controls  that  will  promote 
industry  while  ensuring  that  exported  technology  products  do  not  fall  into  the  wrong  hands. 
According  to  BIS  officials,  export  licenses  are  needed  for  roughly  3%  of  all  exported  products; 
these  are  typically  advanced  technology  products  such  as  lasers,  medicines,  electronics  and 
military  items.  This  list  of  products  is  regularly  updated. 

The  Emerging  Technology  and  Research  Advisory  Committee  (ETRAC)  is  a  Technical 
Advisory  Committee  (TAC)  created  in  2008  (Federal  Register  2008),  whose  role  is  the 
“identification  of  emerging  technologies  and  research  and  development  activities  that  may  be  of 
interest  from  a  dual-use  perspective,  the  prioritization  of  new  and  existing  controls  to  determine 
which  are  of  greatest  consequence  to  national  security,  the  potential  impact  of  dual-use  export 
control  requirements  on  research  activities  and  the  threat  to  national  security  posed  by  the 
unauthorized  export  of  technologies.”35  The  committee  comprises  accomplished  engineers  and 
scientists  from  academia,  industry,  Federal  laboratories,  and  government.  The  Departments  of 
Commerce,  Defense,  State,  and  the  Intelligence  Community  must  be  represented  on  the  committee 
(ETRAC  2012).  The  BIS  Office  of  Technology  Evaluation  (OTE)  administers  ETRAC  and  other 
technical  advisory  committees  and  conducts  in-depth  analysis  of  critical  technologies. 

Process  Overview 

ETRAC  comprises  members  from  across  the  government  and  private  industry  (particularly 
from  sectors  of  interest  such  as  aerospace,  optics,  and  biotechnology).  To  identify  emerging 
technologies,  the  committee  seeks  input  from  experts  from  sensor,  IT,  transportation,  and  other 
advanced  technology  industries  and  visits  relevant  national  laboratories  such  as  the  Anny  Research 
Laboratory  and  the  Air  Force  Research  Laboratory.  ETRAC  also  invites  testimonials  from 
companies  and  looks  at  scientific  publications  and  patents. 

To  understand  the  technology  evolution  and  assess  the  impact  of  the  technology  on  national 
security,  the  committee  looks  to  see  who  is  developing  the  technology  and  whether  there  is  a 
shortage  of  qualified  people.  The  committee  also  does  an  in-depth  analysis  of  the  sale  of  the 
candidate  technologies  and  assesses  the  effect  on  the  industry  if  sales  are  restricted.  ETRAC 
forecasts  3-5  years  out  to  assess  the  impact  of  the  technology  on  national  security  and  economic 
competitiveness.  A  list  of  20  to  30  initial  candidate  technologies  is  reduced  to  6  and  forwarded  for 

33 

BIS  website,  “About  the  Bureau  of  Industry  and  Security,”  http://www.bis.doc.gov/index.php/about-bis. 

34  BIS  website,  “Policy  Guidance,”  https://www.bis.doc.gov/index.php/policy-guidance. 

35  BIS,  TAC  website,  “How  Many  TACs  Are  There  and  What  Do  They  Do?”  http://tac.bis.doc.gov/. 
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action  to  the  Assistant  Secretary  for  Export  Administration.  OTE,  which  reports  to  the  Assistant 
Secretary  for  Export  Administration,  reviews  these  technologies,  along  with  those  that  have  been 
commissioned  by  other  entities,  such  as  government  agencies. 

OTE  examines  each  technology  on  the  list  and  questions  whether  peer  countries  have  similar 
concerns  about  the  technology,  refines  the  assessment  of  where  the  product  is  in  its  evolution,  and 
detennines  the  national  security  implications  of  the  technology.  Export  regulations  are  informed 
by  OTE’s  results,  and  BIS  can  propose  technologies  for  a  multilateral  export  control  regime.36 

No  tools  of  any  kind  are  currently  used  to  identify  emerging  technologies  with  potential  dual- 
use  applications.  The  work  is  done  entirely  by  OTE,  ETRAC  members,  and  the  external  experts 
recruited  by  them. 

Interest  in  Technology  Forecasts 

An  OTE  representative  interviewed  for  this  project  suggested  the  idea  of  a  “red  flag”  tool 
that  would  monitor  the  acquisition  of  companies  internationally  and  alert  an  OSE  analyst  if  a 
certain  number  of  companies  within  a  given  sector  were  purchased  by  companies  in  a  country 
other  than  the  United  States.  This  infonnation  could  support  the  OTE’s  analysis  of  current  foreign 
availability  of  critical  products  and  technologies  and  help  OTE  evaluate  the  expected  impact  of 
export  controls.  The  OTE  representative  said  that  such  a  tool  would  have  been  useful  in  200 1  and 
2002,  when  a  number  of  aerospace  companies  were  purchased  by  the  European  Union.  Rather  than 
focusing  on  gathering  the  data,  the  analyst  could  have  been  focused  on  the  implications  of  these 
purchases  after  being  alerted  about  them. 

Discussion 

Currently,  there  is  no  systematic,  automated  process  to  alert  personnel  at  BIS  about  emerging 
technologies  that  have  the  potential  for  dual-use  applications.  BIS  personnel  rely  on  members  of 
ETRAC  or  other  agencies  and  groups  to  suggest  that  the  OTE  conduct  a  technology  evaluation. 
This  process  means  that  OTE  often  is  evaluating  technologies  in  later  stages  of  development.  A 
technology  forecasting  approach  could  be  used  to  make  OTE  aware  of  a  broader  range  of 
technologies  at  earlier  stages  of  development,  ensuring  more  comprehensive  export  regulation 
considerations.  While  not  discussed  by  any  BIS  members  contacted  for  this  project,  a  tool  that 
determines  the  probability  of  a  technology  being  a  dual-use  technology  could  also  be  useful  to  help 
select  technologies  for  investigation  by  OTE. 


36 

The  United  States  participates  in  four  multilateral  export  control  regimes:  the  Australia  Group,  the  Missile 
Technology  Control  Regime,  the  Nuclear  Suppliers  Group,  and  the  Wassenaar  Arrangement. 
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